Is It Faster to Upload Large Files to Azure Blob or File
in my last post i talked about shared access signature feature of windows azure storage. yous can read that postal service here: http://gauravmantri.com/2013/02/xiii/revisiting-windows-azure-shared-access-signature/ . in this postal service we'll put that to some practical use 
.
1 thing i wanted to achieve recently is the ability to upload very large files into windows azure blob storage from a web awarding . general approach is to read the file through your spider web awarding using " file " html command and upload that unabridged file to some server side lawmaking which would then upload the file in blob storage. this arroyo would work best for smaller files but would fail terribly when it comes to moderately to very large files every bit the file upload control would upload the unabridged file to the server (for bigger files, this would cause timeouts depending on your internet connectedness) and then that file resides in the server retentivity before any action can be taken on that file (again for bigger files, this would cause performance problems assuming y'all take thousands of users uploading thousands of large files).
in this weblog mail service, we'll talk most how we can attain that using file api characteristic in html5 .
please be warned that i'k no javascript/html/css expert . about four – v years ago i used to exercise a lot of javascript development and congenital some really insane applications using javascript (similar post merge kind of features, office automation, native device interfacing all using js) but that was and so. for last four – v years, i accept been extensively doing desktop application evolution. so most of the js code y'all'll see below is copied from various sites and stackoverflow .
objectives
following were some of the objectives i had in mind:
- the interface should be web based (maybe pure html).
- i don't have to share my storage business relationship credentials with the cease users.
- if possible, there should non exist whatever server side code (asp.net/php etc.) i.due east. it should be pure html and the advice should be between user's web browser and storage account.
- if possible, the solution should upload the file in chunks and so that bigger files can be uploaded without reading them in completely in a single get.
now permit's encounter how we can encounter our objectives.
solution
here're some of the things i did to achieve the objectives listed to a higher place.
chunking file
the first challenge i ran into is how to upload the file in chunks. all the examples i saw were uploading an entire file however this is not what i wanted. finally i ran into this case on html5 rocks where they talked about the file api : http://www.html5rocks.com/en/tutorials/file/dndfiles/ . basically what caught my interest there is the " slicing " characteristic available in html v's file interface. in short, what this slice feature does is that it reads a portion of a file asynchronously and returns that data . this is exactly i was looking at. in one case i found that, i knew 75% of my work was done! everything else was a breeze
securing storage account credentials
next matter on my list was securing storage account credentials and this was rather painless every bit i knew exactly what i had to do – shared access signature . sas provided me with a secured uri using which users will exist able to upload files in my storage business relationship without me giving them admission to storage account credentials. this was covered extensively in my post nearly the same: revisiting windows azure shared admission signature .
straight communication between client application and windows azure storage
next thing was to facilitate straight communication between the customer application and storage. as we know windows azure storage is congenital on residual so that means i can only use ajax functionality to communicate with rest api. i important affair to understand is that windows azure storage still does non back up cross-origin resource sharing (cors) at the fourth dimension of writing this blog. what that ways is that your web application and blob storage must be in the same domain. the solution to this trouble is to host your html application in a public blob container in the same storage account where you want your users to upload the files . i've been told that cors support is coming presently in windows azure storage and one time that happens then you need not host this application in that storage account but till then yous would demand to live with this limitation.
the code
now permit's await at the code.
html interface
since i was trying to hack my way through the code, i kept the interface rather elementary. here's how my html code looks like:
<trunk> <class> <div style="margin-left: 20px;"> <h1>file uploader</h1> <p> <strong>sas uri</strong>: <br/> <span class="input-control text"> <input type="text" id="sasurl" fashion="width: 50%" value=""/> </bridge> </p> <p> <strong>file to upload</strong>: <br/> <span class="input-control text"> <input type="file" id="file" name="file" style="width: l%"/> </bridge> </p> <div id="output"> <strong>file properties:</strong> <br/> <p> proper noun: <bridge id="filename"></bridge> </p> <p> file size: <span id="filesize"></span> bytes. </p> <p> file type: <span id="filetype"></span> </p> <p> <input type="button" value="upload file" onclick="uploadfileinblocks()"/> </p> <p> <strong>progress</strong>: <bridge id="fileuploadprogress">0.00 %</span> </p> </div> </div> <div> </div> </form> </body>
all information technology has a textbox for a user to enter sas uri and html file control. once the user selects a file, i display the file properties and the "upload" push button to start uploading the file.
reading file backdrop
to display file properties, i made use of " onchange " outcome of file element. the upshot gave me a list of files. since i was uploading just one file, i picked up the first file from that list and got its name (blob would have that proper noun), size (hulk'due south size and determining clamper size) and type (for setting blob's content type property).
//bind the change event. $("#file").bind('change', handlefileselect); //read the file and detect out how many blocks we would need to split up it. function handlefileselect(e) { var files = e.target.files; selectedfile = files[0]; $("#filename").text(selectedfile.name); $("#filesize").text(selectedfile.size); $("#filetype").text(selectedfile.type); } chunking
in my application i made an assumption that i will split the file in chunks of 256 kb size. once i found the file's size, i merely plant out the full number of chunks.
//read the file and detect out how many blocks nosotros would need to carve up information technology. function handlefileselect(e) { maxblocksize = 256 * 1024; currentfilepointer = 0; totalbytesremaining = 0; var files = e.target.files; selectedfile = files[0]; $("#output").show(); $("#filename").text(selectedfile.name); $("#filesize").text(selectedfile.size); $("#filetype").text(selectedfile.blazon); var filesize = selectedfile.size; if (filesize < maxblocksize) { maxblocksize = filesize; console.log("max block size = " + maxblocksize); } totalbytesremaining = filesize; if (filesize % maxblocksize == 0) { numberofblocks = filesize / maxblocksize; } else { numberofblocks = parseint(filesize / maxblocksize, 10) + 1; } console.log("full blocks = " + numberofblocks); } endpoint for file uploading
the sas uri really represented a uri for blob container. since i had to create an endpoint for uploading file, i split the uri (path and query) and appended the file proper noun to the path and and so re-appended the query to the end.
var baseurl = $("#sasurl").val(); var indexofquerystart = baseurl.indexof("?"); submituri = baseurl.substring(0, indexofquerystart) + '/' + selectedfile.name + baseurl.substring(indexofquerystart); console.log(submituri); reading chunk
this is where file api's chunk function would come in motion picture. what happens in the code is that when the user clicks the upload push button, i read a chunk of that file asynchronously and go a byte array. that byte array will be uploaded.
var reader = new filereader(); var filecontent = selectedfile.slice(currentfilepointer, currentfilepointer + maxblocksize); reader.readasarraybuffer(filecontent);
uploading chunk
since i wanted to implement uploading in clamper, equally soon as a chunk is read from the file, i create a put block request based on put block rest api specification using jquery's ajax function and laissez passer that chunk as data. once this asking is successfully completed, i read the next chunk and repeat the process till the time all chunks are processed.
reader.onloadend = function (evt) { if (evt.target.readystate == filereader.done) { // done == 2 var uri = submituri + '&comp=block&blockid=' + blockids[blockids.length - 1]; var requestdata = new uint8array(evt.target.consequence); $.ajax({ url: uri, type: "put", information: requestdata, processdata: false, beforesend: part(xhr) { xhr.setrequestheader('x-ms-blob-blazon', 'blockblob'); xhr.setrequestheader('content-length', requestdata.length); }, success: office (data, condition) { panel.log(data); console.log(condition); bytesuploaded += requestdata.length; var percentcomplete = ((parsefloat(bytesuploaded) / parsefloat(selectedfile.size)) * 100).tofixed(two); $("#fileuploadprogress").text(percentcomplete + " %"); uploadfileinblocks(); }, error: office(xhr, desc, err) { panel.log(desc); console.log(err); } }); } }; committing hulk
concluding footstep in this process is to commit the blob in blob storage. for this i create a put block list request based on put cake list rest api specification and process that asking over again using jquery's ajax function and laissez passer the cake list equally information. this completed the process.
function commitblocklist() { var uri = submituri + '&comp=blocklist'; console.log(uri); var requestbody = '<?xml version="1.0" encoding="utf-8"?><blocklist>'; for (var i = 0; i < blockids.length; i++) { requestbody += '<latest>' + blockids[i] + '</latest>'; } requestbody += '</blocklist>'; console.log(requestbody); $.ajax({ url: uri, type: "put", information: requestbody, beforesend: function (xhr) { xhr.setrequestheader('x-ms-blob-content-blazon', selectedfile.type); xhr.setrequestheader('content-length', requestbody.length); }, success: function (data, status) { console.log(information); panel.log(status); }, error: office (xhr, desc, err) { console.log(desc); panel.log(err); } }); complete lawmaking
here's the consummate code. for css, i actually used metro ui css – http://metroui.org.ua/ . if you're planning on building web applications and desire to requite them windows 8 applications style wait and experience, exercise give it a try. it's pretty awesome + it's open source. really no reason for yous to not requite it a endeavor!
<!doctype html> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>file uploader</title> <script src="js/jquery-ane.seven.1.js"></script> <link rel="stylesheet" href="css/modernistic.css"/> <script> var maxblocksize = 256 * 1024;//each file will be split in 256 kb. var numberofblocks = 1; var selectedfile = goose egg; var currentfilepointer = 0; var totalbytesremaining = 0; var blockids = new array(); var blockidprefix = "cake-"; var submituri = null; var bytesuploaded = 0; $(certificate).ready(office () { $("#output").hide(); $("#file").bind('change', handlefileselect); if (window.file && window.filereader && window.filelist && window.blob) { // great success! all the file apis are supported. } else { alert('the file apis are not fully supported in this browser.'); } }); //read the file and observe out how many blocks we would need to split it. function handlefileselect(e) { maxblocksize = 256 * 1024; currentfilepointer = 0; totalbytesremaining = 0; var files = e.target.files; selectedfile = files[0]; $("#output").show(); $("#filename").text(selectedfile.name); $("#filesize").text(selectedfile.size); $("#filetype").text(selectedfile.type); var filesize = selectedfile.size; if (filesize < maxblocksize) { maxblocksize = filesize; panel.log("max block size = " + maxblocksize); } totalbytesremaining = filesize; if (filesize % maxblocksize == 0) { numberofblocks = filesize / maxblocksize; } else { numberofblocks = parseint(filesize / maxblocksize, 10) + 1; } panel.log("total blocks = " + numberofblocks); var baseurl = $("#sasurl").val(); var indexofquerystart = baseurl.indexof("?"); submituri = baseurl.substring(0, indexofquerystart) + '/' + selectedfile.name + baseurl.substring(indexofquerystart); console.log(submituri); } var reader = new filereader(); reader.onloadend = part (evt) { if (evt.target.readystate == filereader.done) { // done == two var uri = submituri + '&comp=cake&blockid=' + blockids[blockids.length - i]; var requestdata = new uint8array(evt.target.result); $.ajax({ url: uri, blazon: "put", data: requestdata, processdata: false, beforesend: role(xhr) { xhr.setrequestheader('x-ms-blob-type', 'blockblob'); xhr.setrequestheader('content-length', requestdata.length); }, success: function (data, status) { console.log(data); console.log(status); bytesuploaded += requestdata.length; var percentcomplete = ((parsefloat(bytesuploaded) / parsefloat(selectedfile.size)) * 100).tofixed(2); $("#fileuploadprogress").text(percentcomplete + " %"); uploadfileinblocks(); }, error: function(xhr, desc, err) { panel.log(desc); console.log(err); } }); } }; function uploadfileinblocks() { if (totalbytesremaining > 0) { console.log("current file pointer = " + currentfilepointer + " bytes read = " + maxblocksize); var filecontent = selectedfile.slice(currentfilepointer, currentfilepointer + maxblocksize); var blockid = blockidprefix + pad(blockids.length, half dozen); console.log("block id = " + blockid); blockids.button(btoa(blockid)); reader.readasarraybuffer(filecontent); currentfilepointer += maxblocksize; totalbytesremaining -= maxblocksize; if (totalbytesremaining < maxblocksize) { maxblocksize = totalbytesremaining; } } else { commitblocklist(); } } function commitblocklist() { var uri = submituri + '&comp=blocklist'; console.log(uri); var requestbody = '<?xml version="1.0" encoding="utf-eight"?><blocklist>'; for (var i = 0; i < blockids.length; i++) { requestbody += '<latest>' + blockids[i] + '</latest>'; } requestbody += '</blocklist>'; console.log(requestbody); $.ajax({ url: uri, type: "put", data: requestbody, beforesend: role (xhr) { xhr.setrequestheader('x-ms-hulk-content-blazon', selectedfile.type); xhr.setrequestheader('content-length', requestbody.length); }, success: office (information, status) { console.log(data); console.log(condition); }, fault: function (xhr, desc, err) { console.log(desc); console.log(err); } }); } role pad(number, length) { var str = '' + number; while (str.length < length) { str = '0' + str; } render str; } </script> </head> <body> <form> <div way="margin-left: 20px;"> <h1>file uploader</h1> <p> <strong>sas uri</strong>: <br/> <span class="input-control text"> <input type="text" id="sasurl" style="width: 50%" value=""/> </span> </p> <p> <strong>file to upload</strong>: <br/> <span course="input-command text"> <input type="file" id="file" name="file" style="width: l%"/> </span> </p> <div id="output"> <strong>file properties:</strong> <br/> <p> proper name: <span id="filename"></span> </p> <p> file size: <span id="filesize"></span> bytes. </p> <p> file type: <bridge id="filetype"></span> </p> <p> <input type="button" value="upload file" onclick="uploadfileinblocks()"/> </p> <p> <potent>progress</strong>: <span id="fileuploadprogress">0.00 %</span> </p> </div> </div> <div> </div> </form> </trunk> </html> some caveats
this makes use of html5 file api and while all new browsers support that, same can't exist said about older browsers. if your users would be accessing an awarding like this using older browsers, you would demand to think about alternative approaches. you could either make use of swf file uploader or could write an application using silverlight. steve marx wrote a blog post virtually uploading files using shared access signature and silverlight which you tin read hither: http://web log.smarx.com/posts/uploading-windows-azure-blobs-from-silverlight-part-1-shared-admission-signatures .
i found the code working in ie x , google chrome (version 24.0.1312.57 m) on my windows 8 machine. i got error when i tried to run the lawmaking in firefox (version xviii.0.2) and safari (version v.i.7) browsers so obviously one would demand to keep the browser incompatibility in mind.
enhancements
i hacked this lawmaking in about iv hours or and then and evidently my knowledge is somewhat express when information technology comes to javascript and css so a lot can be improved on that forepart . however some other features i could think of are:
generate sas on demand : you could possibly take a server side component which would generate sas uri on demand instead of having a user enter that manually.
multiple file uploads : this awarding can certainly exist extended to upload multiple files. a user would select multiple files (or may exist even a binder) and accept the application upload multiple files.
drag/drop support : this application can certainly be extended to support drag/drop scenario where users could elevate files from their desktop and drop them to upload.
do upload in web worker : this is another comeback that can be done where uploads are washed through web worker capability in html5.
parallel uploads : currently the lawmaking uploads one clamper at a fourth dimension. a modification could be to upload multiple chunks simultaneously.
transient fault treatment : since windows azure storage is a remote shared resource, you may encounter transient errors. you could change the application to handle these transient errors. for more details on transient errors, please come across this web log mail service of mine: http://gauravmantri.com/2013/01/11/some-best-practices-for-building-windows-azure-deject-applications/ .
summary
so that's it for this post! every bit you saw, it is quite like shooting fish in a barrel to implement a very simple html/js based application for getting data into windows azure hulk storage. patently at that place're some limitations and there's cross-browser compatibility problems one would need to consider but once those are sorted out, it opens upwardly a lot of heady opportunities. i hope y'all've found this postal service useful. as always, if you find any problems with the post please let me know and i'll fix it asap.
happy coding!
Source: https://dzone.com/articles/uploading-large-files-windows
0 Response to "Is It Faster to Upload Large Files to Azure Blob or File"
Post a Comment