Re: File upload stats in CP 3.0

Just wanted to let you know I figured out something. If you
include two
classes in your server to basically override the file object
that cherrypy
uses by default, create a dictionary of file uploads in the
cherrypy module,
and then add a function to track stats, you can get it to
work. As such...:

Add these two classes:
class FieldStorage(cherrypy._cpcgifs.FieldStorage):
    ''' We want control over our timing and download
status,
        so we've got to override the original. This will
work
        transparently without interfering with the user,
but
        might warrant addition to _cpcgifs '''

    def __del__(self, *args, **kwargs):
        try:
            dcopy =
cherrypy.file_transfers[cherrypy.request.remote.ip].copy()
            for key, val in dcopy.iteritems():
                if val.transfered == True:
                    del
cherrypy.file_transfers[cherrypy.request.remote.ip][key]
            del dcopy
            if
len(cherrypy.file_transfers[cherrypy.request.remote.ip]) ==
0:
                del
cherrypy.file_transfers[cherrypy.request.remote.ip]

        except KeyError:
            pass

    def make_file(self, binary=None):
        fo = ProgressFile(self.bufsize)
        if
cherrypy.file_transfers.has_key(cherrypy.request.remote.ip):

           
cherrypy.file_transfers[cherrypy.request.remote.ip]
                    [self.filename] = fo
        else:
           
cherrypy.file_transfers[cherrypy.request.remote.ip]
                    = {self.filename:fo}

        return fo

class ProgressFile(object):
    def __init__(self, buf, *args, **kwargs):
        self.file_object = tempfile.TemporaryFile(*args,
**kwargs)
        self.transfered = 0
        self.buf = buf
        self.pre_sized =
float(cherrypy.request.headers['Content-length'])
        self.speed = 1
        self.remaining = 0
        self.eta = 0
        self._start = time.time()

    def write(self, data):
        now = time.time()
        self.transfered += len(data)
        upload_timeout = getattr(cherrypy.thread_data,
'upload_timeout',
False)
        if upload_timeout:
            if (now - self._start) > upload_timeout:
                raise Upload_TimeoutError

        upload_maxsize = getattr(cherrypy.thread_data,
'upload_maxsize',
False)
        if upload_maxsize:
            if self.transfered > upload_maxsize:
                raise Upload_MaxSizeError

        self.speed = self.transfered / (now - self._start)

        upload_minspeed = getattr(cherrypy.thread_data,
'upload_minspeed',
False)
        if upload_minspeed:
            if self.transfered > (5 * self.buf): # gives
us a reasonable
wait period.
                if self.speed < upload_minspeed:
                    raise Upload_UpSpeedError

        self.remaining = self.pre_sized - self.transfered

        if self.speed == 0: self.eta = 9999999
        else: self.eta = self.remaining / self.speed

        return self.file_object.write(data)

    def seek(self, pos):
        self.post_sized = self.transfered
        self.transfered = True
        return self.file_object.seek(pos)

    def read(self, size):
        return self.file_object.read(size)

Then be sure to add this line at the end:
cherrypy._cpcgifs.FieldStorage = FieldStorage

then here's what I used to get upload stats:
cherrypy.expose
   def upload_stats(self, *args, **kwargs):
      try:
         stat =
cherrypy.file_transfers[cherrypy.request.remote.ip]
         # Convert everything to KBs and return
         for key, val in stat.iteritems():
            speed = '%9.2f' % (val.speed / 1024.0)
            total = '%9.2f' % (val.pre_sized / 1024.0)
            transfered = '%9.2f' % (val.transfered /
1024.0)
            eta = str(int(val.eta))
            
            done = val.transfered / val.pre_sized
            done = str(300 * done)
            Logger.info("Upload Stats: fn: %s speed: %s
total: %s
transferred: %s" % (key, speed, total, transfered))
            yield json.write({'filename':key, 'speed':
speed, 'total':total,
'transfered':transfered, 'eta': eta, 'done':done, 'status':
'uploading'})
      except KeyError:
         Logger.info("Upload Stats: returning
done")
         # If there are no entries by our IP, then we have
nothing.
         yield json.write({'status':'done'})

After that, you can just implement your uploads like normal,
but now you can
track the individual file progress based on the user's IP
address (they key
for the file in the file dictionary). Hope this helps
someone

-Brett
-- 
View this message in context: http://www.nabble.com/File-upload-stat
s-in-CP-3.0-tf4721728.html#a13596800
Sent from the cherrypy-users mailing list archive at
Nabble.com.


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "cherrypy-users" group.
To post to this group, send email to cherrypy-usersgooglegroups.com
To unsubscribe from this group, send email to
cherrypy-users-unsubscribegooglegroups.com
For more options, visit this group at h
ttp://groups.google.com/group/cherrypy-users?hl=en
-~----------~----~----~----~------~----~------~--~---

hctv19 wrote:
> Just wanted to let you know I figured out something. If
you include
two
> classes in your server to basically override the file
object that
> cherrypy uses by default, create a dictionary of file
uploads in
> the cherrypy module, and then add a function to track
stats, you can
> get it to work. As such...:

Great work! It would be nice to have this at http://tools.cherrypy.org.



> Add these two classes:
> class FieldStorage(cherrypy._cpcgifs.FieldStorage):
>     ''' We want control over our timing and download
status,
>         so we've got to override the original. This
will work
>         transparently without interfering with the
user, but
>         might warrant addition to _cpcgifs '''
> ...
> Then be sure to add this line at the end:
> cherrypy._cpcgifs.FieldStorage = FieldStorage

That shouldn't really go into _cpcgifs itself, because it's
a bit too
much overhead for those who don't need it. However, it
should probably
be easier to override the builtin FieldStorage class
without
monkeypatching. If we changed
_cprequest.Request.process_body to read:

        # FieldStorage only recognizes POST, so fake it.
        methenv = {'REQUEST_METHOD': "POST"}
        try:
            forms = self.FieldStorage(fp=self.rfile,
                                      headers=self.headers,
                                      environ=methenv,
                                      keep_blank_values=1)
        except http.MaxSizeExceeded:
            # Post data is too big
            raise cherrypy.HTTPError(413)

...and have Request.FieldStorage default to
_cpcgifs.FieldStorage, then
you could attach your custom FieldStorage class in config
with
"request.FieldStorage = my.StatFieldStorage". Make
me a ticket and I'll
see if we can get it into 3.1 final.


Robert Brewer
fumanchuaminus.org

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the
Google Groups "cherrypy-users" group.
To post to this group, send email to cherrypy-usersgooglegroups.com
To unsubscribe from this group, send email to
cherrypy-users-unsubscribegooglegroups.com
For more options, visit this group at h
ttp://groups.google.com/group/cherrypy-users?hl=en
-~----------~----~----~----~------~----~------~--~---