Third Light Developer Exchange

mdyason

Member

Registered: 2015-01-20

Posts: 43

Asynchronous upload limit.

We are transferring large numbers of files to our IMS and create many uploads, monitor them and when they are all complete we call Upload.GetUploadResult to get the IMS reference numbers for our files.  This works fine to a point.  There seems to be a limit to how many files we can add to each upload.  So uploading the same 62 files using 2 uploads works fine if we put 40 files in the first upload and 22 files in the second one.  However if we put 60 in the first upload and 2 in the second upload, the upload does not fail (All 60 images do go into the system), but the API tells us that 0 files were uploaded.

Typically the reply from this call looks like: 
{"result":{"api":"OK","action":"OK","timings":{"initTime":52,"setupTime":8,"methodTime":9,"totalTime":69}},"outParams":{"succeeded":
{"132957-16282-2013-04-21 19:10:10-IMG_0191.JPG":"37412142170"
,"132957-16282-2013-04-21 19:10:14-IMG_0192.JPG":"37412142760"}
,"failed":[],"error":false,"count":2}}

However for the 60 files in the above example it looks like this:
{"result":{"api":"OK","action":"OK","timings":{"initTime":56,"setupTime":8,"methodTime":9,"totalTime":73}},"outParams":{"succeeded":[],"failed":[],"error":false,"count":0}}

Can you offer an explanation of why this happens?  What parameters are coming into play here?  (Remember that all 60 files do get uploaded)

long ago we had the number per upload at 100 and this worked for awhile.  When it failed we made the number 60 and that worked for awhile.  When it failed again we made it 40.
I originally thought it may be something to do with actual file size, but I am not convinced.

ben

Third Light Staff

From: Third Light

Registered: 2013-06-06

Posts: 79

Re: Asynchronous upload limit.

Hi mdyason,

We're looking in to this part of the codebase to investigate what might be at play but initial thoughts are that there shouldn't be any limit to the number of files. Basically this seems on the face of it to be a bug.

To help our investigation please could you let us know the method calls you are making to process an upload of 60 files? I'm particularly interested in:
- the parameters to CreateUpload
- the method you are using to transfer the file data (i.e the postUploadUrl or base64Encoded in an API call)
- are you sending all the files at once in one large transaction, or individually with multiple transactions
- how you are testing the progress of the uploaded files
- and any calls you make at the end

Also, roughly how long from start to finish does it take? So how long to transfer the files, and then how long for the upload job to process the files. (appreciate that this might be hard to determine especially considering the issue you're reporting!)

Many thanks,
Ben

mdyason

Member

Registered: 2015-01-20

Posts: 43

Re: Asynchronous upload limit.

Hi Ben

Below i will lay out the steps taken by our software and show the code we use (c#).  Hopefully this will give you a complete picture.

As far as time is concerned.  I have not actually timed this but subjectively I would guess it is less than 10 minutes for the whole job of 62 files originally mentioned.
There is a delay between finding that the first set of files has gone up and reading the upload results.  This is because we do both uploads and get 100% from both uploads before we read the upload results for each upload.  In the case we are currently discussing, that delay is probably not a problem but it may be, in the case where we have hundreds of uploads all running at the same time.  In that case we may wait 12 or more hours to read the upload results for the first upload.  As far as the system actually processing the images is concerned - for small uploads like the 62 we are talking about, I have never noticed any sort of delay.  When the upload is finished, the system has them and behaves normally.  When we upload say 20 thousand images in one go, the system will be a bit unresponsive for about a day. (But only in that one folder)

For a set of 60 (or 40) files.  (These files all exist in a single folder on a networked hard drive - we have already checked to make sure they are all valid image files)

1:  convert each file into a Base64String (Filestring)

2:  for each file, build a metadata dictionary (metadataDict)
Looks like:

	{"siteCode", image.SiteCode},
	{"workTypeName", image.WorkTypeName},
	{"workTypeCode", image.WorkTypeCode},
	{"projectFinancialCode", image.ProjectFinancialCode},
	Etc.

3:  for set of files, build a dictionary (imageInfoDict)
Looks like:   

	{"encoding", "base64"},
	{"name", TL_ImageName},
	{"data", fileString},
	{"metadata", metadataDict}

4:  create upload and get an upload key (uploadKey)

	{
	action = "Upload.CreateUpload",
	apiVersion = "1.0",
	sessionId = _sessionId,
	inParams = new Dictionary<string, object>
		{
			{
				"params", new Dictionary<string, object>
				{
					{"destination", _folderId},
					{"synchronous", false},
					{"lifetime", "86400"},
					{"lifetimemax", "250000"},
					{"lifetimeincrement", "86400"}
				}
			}
		}
	}

5:  add the whole set of files to upload in a single call

	{
	action = "Upload.AddFilesToUpload",
	apiVersion = "1.0",
	sessionId = _sessionId,
	inParams = new Dictionary<string, object>
		{
			{"uploadKey", uploadKey},
			{"fileData", _fileData}
		}
	}

6:  start upload

	{
	action = "Upload.StartUpload",
	sessionId = _sessionId,
	apiVersion = "1.0.1",
	inParams = new Dictionary<string, string> { { "uploadKey", uploadKey } }
	}

7:  monitor upload until it returns 100 percent (Try every 3 seconds)

	{
	action = "Upload.GetUploadProgress",
	apiVersion = "1.0",
	sessionId = _sessionId,
	inParams = new Dictionary<string, string> { { "uploadKey", uploadKey } }
	}

8:  get upload results

	{
	action = "Upload.GetUploadResult",
	sessionId = _sessionId,
	apiVersion = "1.0.1",
	inParams = new Dictionary<string, string> { { "uploadKey", uploadKey } }
	}

ben

Third Light Staff

From: Third Light

Registered: 2013-06-06

Posts: 79

Re: Asynchronous upload limit.

Hi,
Many thanks for your very detailed reply. That has helped enormously to understand the steps you are taking. I have an idea what might be going on - could I ask roughly what bytesize of metadata would be added for each file. i.e. sum the bytes of text information? Appreciate this might vary per file but a rough average would be useful.  I think the storage in the database for the upload job might not be optimal and you are hitting a content-length limit in a database field.

Many thanks,
Ben

mdyason

Member

Registered: 2015-01-20

Posts: 43

Re: Asynchronous upload limit.

Hi Ben

The metadata on each file is:

var metadataDict = new Dictionary<string, object>
{
	{"siteCode", image.SiteCode},
	{"workTypeName", image.WorkTypeName},
	{"workTypeCode", image.WorkTypeCode},
	{"projectFinancialCode", image.ProjectFinancialCode},
	{"projectName", image.ProjectName},
	{"marketSegment", image.MarketSegment},
	{"clientName", image.ClientName},
	{"serviceLine", image.ServiceLine},
	{"origin", image.Origin},
	{"originalFileName", image.FileName},
	{"imageCategory", "Project"},
	{"fieldKeywords", Keywords},
	{"processWorkflow", image.ProcessWorkflow},
	{"description", image.Description}
}; 

The only fields here that are longer than about 15 characters are:
  The client or project name which may be up to 120 characters each. 
  The description is typically empty but theoretically could be a sentence or two.
  The origin which is the path on the hard drive (max 260 characters)
  The keywords which are obtained by splitting up the path (max 260 characters) into a number of discrete words (typically 10 to 15 keywords)

Note that the same metadata is attached to every image in the upload.

ben

Third Light Staff

From: Third Light

Registered: 2013-06-06

Posts: 79

Re: Asynchronous upload limit.

Hi again,
Sorry for the delay - thanks again for the information. Just to say that we're testing out some theories and will update you soon.
Best wishes,
Ben

ben

Third Light Staff

From: Third Light

Registered: 2013-06-06

Posts: 79

Re: Asynchronous upload limit.

Hi again,
We've fixed the bug here and it is going through QA before being made available as a Release Candidate. We'll let you know over the support channel when it will be available to you to install.
Thanks again for your detailed replies.
Best wishes,
Ben