python 3.x - kinesis firehose how to change filename and folder hierarchy

Question

I have to upload millions of images on s3. the bottleneck is rate limit of s3 which is less then 3500 per second
so I decided to try Kinesis Firehose, but stream accept Blob and it upload to S3 with uuid name and date prefix.
What I actually want to do is .

• custom directory structure for each file
• custom name for each image is there any way to-do this .

Answer 1

In short, no. You can not control output file name except for a prefix (see documentation here

I think you should re-think how you want to process those records as they come out. FireHose batches the incoming data so it wouldn't make sense to split things out based upon filename as it comes in, because in your case multiple images(data) would be within one record that Kinesis FireHose writes out.

If you need to separate out records as they get processed by FireHose, you would have to read in each batch of records that gets written out (which would be all within one S3 object), then you could put them anywhere you want with a lambda or something else similar.