How to augment in real world using QRCode

There are several ways to augment content in the real world at user-specified positions either through image tracking or spatial anchoring using ARKit/ARCore (Unity ARFoundation).

But there are certain limitations with image tracking such as — images should have enough features, images need to be in appropriate size (in Mb) and shape (aspect ratio) for it to be accepted, pre-defined image/QRCode set must be available for tracking etc. Also, external factors like lighting conditions or any physical changes in the real environment affect the image tracking or spatial anchoring. It also requires a separate backend service to store the spatial information for anchoring.

Another quick way is to use QRCode to augment content in the real world. Here the QRCode holds the information we want it to hold and also acts are the pose identifier to augment in the real world. Identifying the QRCode pose in the real world could be done though OpenCV as well. But here we will use Unity ARFoundation blended with ZXing QRCode library to augment around the QRCode

1. Download the sample ARFoundation project with components as shown in the snapshot below. Import the ZXing dll library for Unity into the project. Make sure you have the ARPointCloudManager or the ARPlaneManager on the AR Session Origin object as shown in the snapshot below. Create a new monobehavior script and attach it to an empty game object. This will start the ARCamera and start placing AR point cloud in the environment. Use the below code snippet in the new script:

Note: You can change the alpha for the point cloud prefab if you don’t want the feature points to be visible in the real world.

Default Hierarchy on an ARFoundation Sample Scene

Inspector View for AR Session Origin

Code Snippet :

using ZXing;
public GameObject arObjectOnQRCode; //Reference to the AR Gameobject which needs to be placed on scanning the QRCode
IBarcodeReader reader; //QRCode reading library
ARCameraManager aRCamera;
ARRaycastManager arRaycastManager;
private Texture2D arCameraTexture; //Texture to hold the processed AR Camera frame
private bool onlyonce;
Void Start(){
aRCamera = FindObjectOfType<ARCameraManager>(); //Load the ARCamera
arRaycastManager = FindObjectOfType<ARRaycastManager>(); //Load the Raycast Manager
//Get the ZXing Barcode/QRCode reader
reader = new BarcodeReader();
//Subscribe to read AR camera frames: Make sure this statement runs only once
aRCamera.frameReceived += OnCameraFrameReceived;
}

2. Get the AR Camera frames and send them to a coroutine to decode information from the scanned QRCode.

unsafe void OnCameraFrameReceived(ARCameraFrameEventArgs eventArgs){
if((Time.frameCount % 15) == 0){ //You can set this number based on the frequency to scan the QRCode
XRCameraImage image;
if (aRCamera.TryGetLatestImage(out image)){
StartCoroutine(ProcessQRCode(image));
image.Dispose();
}}}

3. Process the frame and look for a QRCode. Use the hit position on the QRCode to augment the object.

//Asynchronously Convert to Grayscale and Color : https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@1.0/manual/cpu-camera-image.html
IEnumerator ProcessQRCode(XRCameraImage image){
// Create the async conversion request
var request = image.ConvertAsync(new XRCameraImageConversionParams{
inputRect = new RectInt(0, 0, image.width, image.height),
outputDimensions = new Vector2Int(image.width / 2, image.height / 2),
// Color image format
outputFormat = TextureFormat.RGB24,
// Flip across the Y axis
//  transformation = CameraImageTransformation.MirrorY
});
while (!request.status.IsDone())
yield return null;
// Check status to see if it completed successfully.
if (request.status != AsyncCameraImageConversionStatus.Ready){
// Something went wrong
Debug.LogErrorFormat("Request failed with status {0}", request.status);
// Dispose even if there is an error.
request.Dispose();
yield break;}
// Image data is ready. Let's apply it to a Texture2D.
var rawData = request.GetData<byte>();
// Create a texture if necessary
if (arCameraTexture == null){
arCameraTexture = new Texture2D(
request.conversionParams.outputDimensions.x,
request.conversionParams.outputDimensions.y,
request.conversionParams.outputFormat,
false);
}
// Copy the image data into the texture
arCameraTexture.LoadRawTextureData(rawData);
arCameraTexture.Apply();
byte[] barcodeBitmap = arCameraTexture.GetRawTextureData();
LuminanceSource source = new RGBLuminanceSource(barcodeBitmap,arCameraTexture.width, arCameraTexture.height);
//Send the source to decode the QRCode using ZXing
if (!onlyonce){ //Check if a frame is already being decoded for QRCode. If not, get inside the block.
doOnce= true; //Now frame is being decoded for a QRCode
//decode QR Code
result = reader.Decode(source);
if (result != null && result.Text != ""){ //If QRCode found inside the frame
QRContents = result.Text;
// Get the resultsPoints of each qr code contain the following points in the following order: index 0: bottomLeft index 1: topLeft index 2: topRight
//Note this depends on the oreintation of the QRCode. The below part is mainly finding the mid of the QRCode using result points and making a raycast hit from that pose.
ResultPoint[] resultPoints = result.ResultPoints;
ResultPoint a = resultPoints[1];
ResultPoint b = resultPoints[2];
ResultPoint c = resultPoints[0];
Vector2 pos1 = new Vector2((float)a.X, (float)a.Y);
Vector2 pos2 = new Vector2((float)b.X, (float)b.Y);
Vector2 pos3 = new Vector2((float)c.X, (float)c.Y);
Vector2 pos4 = new Vector2(((float)b.X - (float)a.X) / 2.0f, ((float)c.Y - (float)a.Y) / 2.0f);
List<ARRaycastHit> aRRaycastHits = new List<ARRaycastHit>();
//Make a raycast hit to get the pose of the QRCode detected to place an object around it.
if (arRaycastManager.Raycast(new Vector2(pos4.x, pos4.y), aRRaycastHits, TrackableType.FeaturePoint) && aRRaycastHits.Count > 0){
//To shift the object to a relative position by adding/subtracting a delta value, uncomment the below line.
//Instantiate an object at Hitpose found on the QRCode
GameObject NewObjectToPlace = Instantiate(arObjectOnQRCode, aRRaycastHits[0].pose.position);
//OR
// Use default position to place the object in front of the camera if the Hit Pose is not found. //You can uncomment the below code for this default behaviour
//defaultObjectPosition = Camera.main.ScreenToWorldPoint(new Vector3(Screen.width / 2, //Screen.height / 2, Camera.main.nearClipPlane));
//OR
//Reposition the Augmented Object by adding some delta
//NewObjectToPlace.transform.position = new //Vector3(NewObjectToPlace.transform.position.x + xDelta, //NewObjectToPlace.transform.position.y, NewObjectToPlace.transform.position.z);
}
else{
doOnce= false; //Continue processing the next frame to decoded for a QRCode if hit not found
}}
else{
doOnce= false;  //QRCode not found in the frame. Continue processing next frame for QRCode
}}}

Things you would want to consider:

You can also try to make a Raycast hit from the screen centre -

if (arRaycastManager.Raycast(new Vector2(Screen.width / 2, Screen.height / 2), aRRaycastHits, TrackableType.FeaturePoint) && aRRaycastHits.Count > 0)

You can also use All trackable objects or AR Plane to make a hit.

To conclude, using the QRCode to augment and view content relieves user from any environment changes like external lighting affecting image tracking or reshuffling the real objects in the environment affecting spatial anchoring.

QRCode for augmentation can be applied to use cases where you have dynamic environment, changing lighting conditions, limited bandwidth or for quick experimentation with AR. Augmenting telemetry information for the machines or industrial devices tagged with QRCode inside a manufacturing industry is one good application.