No response is received when tried to upload 800MB file.

satishd · 2015-05-21 15:48:06

Hi There,

I am unable to upload file to third light as it doenot respond back whether the process is working or failed, waited for more than couple of
hours still waiting for the response. we did raise the ticket with support team in last December and were told there was code issue within third light and which was fixed. but we are facing again same issue where file is not getting uploaded

the following is createURL() request is being sent, please let me know where it is going wrong, do you think the parameters passed by us is good? . it was working fine until few days ago, is it better to restart our web instance ?

json = "{\"action\":\"Upload.CreateUpload\"," +
"\"apiVersion\":\"1.0\"," +
"\"inParams\":{\"params\":{\"destination\":\"" + parentId + "\"," +
"\"synchronous\":false," +
"\"allowotheruserupload\":true," +
"\"editablemetadata\":{\"caption\":\"OPTIONAL\",\"keywords\":\"OPTIONAL\",\"source\":\"OPTIONAL\"}," +
"\"lifetime\":3600}" +
"}," +
"\"sessionId\":\"" + sessionId + "\"" +
"}";

Thanks
Satish

ben · 2015-05-21 16:04:48

Hi Satish,
Please could you clarify which request is resulting in a very long wait? Perhaps you could post a transcript of the entire transaction from login to the file upload?

Many thanks,
Ben

satishd · 2015-05-21 16:19:47

We are facing delay after file upload to the following URL- once the content is uploaded we're waiting for the response, which is causing an issue.

https://mysite.thirdlight.com/dofileupl … t1gFtJh1MB

please find the entire method snippet.

please find the code where the we are facing when we tried to read the response from Webrequest.

try
{

string boundary = "---------------------------" + DateTime.Now.Ticks.ToString("x");
byte[] boundarybytes = System.Text.Encoding.ASCII.GetBytes("\r\n--" + boundary + "\r\n");

HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(urlToUse + "&t=ajax");
wr.ContentType = "multipart/form-data; boundary=" + boundary;
wr.Method = "POST";
wr.KeepAlive = true;
wr.Timeout = System.Threading.Timeout.Infinite;
wr.Headers.Add("IMSSESSID", sessionId);

Stream rs = wr.GetRequestStream();

string formdataTemplate = "Content-Disposition: form-data; name=\"{0}\"\r\n\r\n{1}\r\n";
foreach (string key in nvc.Keys)
{
rs.Write(boundarybytes, 0, boundarybytes.Length);
string formitem = string.Format(formdataTemplate, key, nvc[key]);
byte[] formitembytes = System.Text.Encoding.UTF8.GetBytes(formitem);
rs.Write(formitembytes, 0, formitembytes.Length);
}
rs.Write(boundarybytes, 0, boundarybytes.Length);

string headerTemplate = "Content-Disposition: form-data; name=\"{0}\"; filename=\"{1}\"\r\nContent-Type: {2}\r\n\r\n";
string header = string.Format(headerTemplate, "file", Path.GetFileName(fileUrl), Path.GetExtension(fileUrl));
byte[] headerbytes = System.Text.Encoding.UTF8.GetBytes(header);
rs.Write(headerbytes, 0, headerbytes.Length);

FileStream fileStream = new FileStream(fileUrl, FileMode.Open, FileAccess.Read);

byte[] buffer = new byte[4096];
int bytesRead = 0;
while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
{
rs.Write(buffer, 0, bytesRead);
}
fileStream.Close();

byte[] trailer = System.Text.Encoding.ASCII.GetBytes("\r\n--" + boundary + "--\r\n");
rs.Write(trailer, 0, trailer.Length);
rs.Close();

WebResponse wresp = null;
try
{
wresp = wr.GetResponse(); *******NO RESPONSE IN THIS LINE... IT JUST HUNGS UP.****
Stream stream2 = wresp.GetResponseStream();
StreamReader reader2 = new StreamReader(stream2);
result = reader2.ReadToEnd();

if (!string.IsNullOrEmpty(result))
{
JObject joResponse;
joResponse = JObject.Parse(result);
return Convert.ToBoolean(joResponse["error"]);
}

reader2.Close();
stream2.Close();
wresp.Close();

}
catch (Exception ex)

.....

ben · 2015-05-21 16:41:53

Thanks Satish,
I can't see anything that stands out so I will do some investigating and get back to you.
In the mean time could you let me know if the files make it into IMS eventually? And is it just large files that cause a hang? Does a small JPG upload fine for example?

Cheers
B

satishd · 2015-05-21 16:48:31

Thanks Ben,

Small files are uploading fine. it is just this large file getting struck.

Regards
Satish

ben · 2015-05-21 17:58:49

Hi again,
I can't find anything wrong. With those upload settings, the POST request is just transferring the file into temporary storage on the IMS server - there is no post processing happening at that stage so there should be no significant delay in returning a response. You must be calling StartUpload at some point further on in the script in order to process the files and import them into the system (please could you confirm this?).

So can I just ask you to check that the script is definitely hanging at the point you say - i.e the bit where it's waiting for a response. The long wait might be the time it takes to transfer the 800MB file to our servers and so it would be spending time in this section:

                while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
                {
                    rs.Write(buffer, 0, bytesRead);
                }

Do you know your outbound bandwidth? You can calculate the expected time to transfer from this.

An alternative theory is that you are transferring through a network proxy and that is interfering with the http post request such that the response never comes back. I have to say that I'm not really convinced by this explanation - if there was a proxy problem I would expect the connection to be closed early rather than never at all.

Regards,
Ben

satishd · 2015-05-22 09:48:55

Hi Ben,

Yes you are right, I'm calling StartUpload after the above post request to make sure the above process returns TRUE, is this unnecessary step?

json = "{\"action\":\"Upload.StartUpload\"," +
"\"apiVersion\":\"1.0\"," +
"\"sessionId\":\"" + sessionId + "\"," +
"\"inParams\":{\"uploadKey\":\"" + uploadKey + "\"" +
"}" +
"}";

Thanks
Satish

Last edited by satishd (2015-05-22 09:49:18)

satishd · 2015-05-22 09:51:38

Hi Ben,

The process is not spending time on following section,

while ((bytesRead = fileStream.Read(buffer, 0, buffer.Length)) != 0)
{
rs.Write(buffer, 0, bytesRead);
}

but it is getting hung up at following line.

try
{
wresp = wr.GetResponse(); *******NO RESPONSE IN THIS LINE... IT JUST HUNGS UP.****
Stream stream2 = wresp.GetResponseStream();
StreamReader reader2 = new StreamReader(stream2);
result = reader2.ReadToEnd();

if (!string.IsNullOrEmpty(result))
{
JObject joResponse;
joResponse = JObject.Parse(result);
return Convert.ToBoolean(joResponse["error"]);
}

reader2.Close();
stream2.Close();
wresp.Close();

}
catch (Exception ex)
...

satishd · 2015-05-22 10:46:26

Hi Ben,
our network outbound bandwidth is 50MBPS, so it should not take more than 20-25minutes to upload such a large file.
thanks
Satish

ben · 2015-05-22 11:27:30

Hi Satish,
OK, fair enough. Can we arrange a time when you kick off the script and I'll watch on the server? How about 3pm today?
Ben

satishd · 2015-05-22 11:47:18

Sure, will start running my process at 3pm.

satishd · 2015-05-22 15:04:42

Hi Ben,

I have just run now, waiting for the response.

Thanks
Satish

ben · 2015-05-22 15:21:30

Hi Satish,
Great, whilst that was happening I saw a tmp file filling up - I saw it at 125Mbytes before it got moved into normal storage so it could have been bigger. And then apache issued a response shortly after:

212.250.100.254 - - [22/May/2015:15:02:15 +0100] "POST /dofileupload.tlxr?task=7lxc7lxyuMA7i1N7QMhra&t=ajax HTTP/1.1" 200 16

Can you confirm whether 212.250.100.254 is your IP address? And what size the file was.

In you IMS site I can see a recently uploaded mp3 file that seems to match the upload date and time. Reference number 29685828320.

Best wishes,
Ben

satishd · 2015-05-22 15:33:56

Still waiting for the response, anything found from your end ?

ben · 2015-05-22 15:37:25

I think you may have missed my reply at 3.20pm - pretty sure I've saw your request in the logs and apache has responded. It looks to be an issue your end - perhaps your router or a proxy is dropping the request for some reason.

satishd · 2015-05-22 15:43:02

First I have uploaded 175MB file, by mistakenly which went through fine, the second request is for 800MB which has stopped responding..
can I restart the process now just to concentrate on this file?>

satishd · 2015-05-22 15:45:58

yes Ben, you are correct the IP address is 212.250.100.254

ben · 2015-05-22 16:07:58

Hi Satish, I was busy with something else. Yes please try again now if you wish. Ben

satishd · 2015-05-22 16:23:13

Hi Ben,

Something strange happening , now the response is coming back after couple of minutes, any change done from your side ?

Thanks
satish

ben · 2015-05-22 16:26:40

I saw your requests - all looked to be normal. I haven't changed anything I'm afraid - although I guess this is a good thing!

satishd · 2015-05-22 16:31:56

I could upload two files in sequence, all working fine... without any changes from our network side or code. Only thing is now I have updated fiddler on my system to monitor the traffic, will that anyway waking up http request/respnse actions?

ben · 2015-05-22 16:39:54

So it now works fine, but we're not quite sure why?!

satishd · 2015-05-22 16:47:46

Thanks Ben.

Third Light Developer Exchange

Announcement

If you wish to join the Developer Exchange, please contact your account manager - this is to avoid unnecessary spam in the forums. Many thanks for your understanding.

#1 2015-05-21 15:48:06

No response is received when tried to upload 800MB file.

#2 2015-05-21 16:04:48

Re: No response is received when tried to upload 800MB file.

#3 2015-05-21 16:19:47

Re: No response is received when tried to upload 800MB file.

#4 2015-05-21 16:41:53

Re: No response is received when tried to upload 800MB file.

#5 2015-05-21 16:48:31

Re: No response is received when tried to upload 800MB file.

#6 2015-05-21 17:58:49

Re: No response is received when tried to upload 800MB file.

#7 2015-05-22 09:48:55

Re: No response is received when tried to upload 800MB file.

#8 2015-05-22 09:51:38

Re: No response is received when tried to upload 800MB file.

#9 2015-05-22 10:46:26

Re: No response is received when tried to upload 800MB file.

#10 2015-05-22 11:27:30

Re: No response is received when tried to upload 800MB file.

#11 2015-05-22 11:47:18

Re: No response is received when tried to upload 800MB file.

#12 2015-05-22 15:04:42

Re: No response is received when tried to upload 800MB file.

#13 2015-05-22 15:21:30

Re: No response is received when tried to upload 800MB file.

#14 2015-05-22 15:33:56

Re: No response is received when tried to upload 800MB file.

#15 2015-05-22 15:37:25

Re: No response is received when tried to upload 800MB file.

#16 2015-05-22 15:43:02

Re: No response is received when tried to upload 800MB file.

#17 2015-05-22 15:45:58

Re: No response is received when tried to upload 800MB file.

#18 2015-05-22 16:07:58

Re: No response is received when tried to upload 800MB file.

#19 2015-05-22 16:23:13

Re: No response is received when tried to upload 800MB file.

#20 2015-05-22 16:26:40

Re: No response is received when tried to upload 800MB file.

#21 2015-05-22 16:31:56

Re: No response is received when tried to upload 800MB file.

#22 2015-05-22 16:39:54

Re: No response is received when tried to upload 800MB file.

#23 2015-05-22 16:47:46

Re: No response is received when tried to upload 800MB file.

Board footer