3

I'm in the process of writing an image analysis pipeline and I've hit a bit of a roadblock. Here's the 30,000-ft view:

  1. Images arrive as base64-encoded PNG images over standard input
  2. Images are decoded, and processed. I don't think the details are particularly relevant, but I am happy to offer specifics in the form of an edit.
  3. When the analysis is complete, the entire stream must be concatenated into a seekable, timestamped format (that is, a video file of some sort) and saved to disk.

My question pertains to stage 3. At this final stage of the pipeline, each frame will have a set of associated features that were extracted via the image analysis performed in stage 2. I would like for the final output format (the video format) to preserve this information alongside the frame that "owns" it. In other words, one should think about this as a separate channel that evolves alongside the video data, locked to the same time reference. This would notably allow me to treat features as time-series.

Here's a simple diagram to further illustrate what I'm trying to achieve.

-0 ms--16 ms--32 ms--48 ms--64 ms- ... n ms ->  # time channel
-F1----F2-----F3-----F4-----F5---- ... Fn --->  # video channel
-1.1---3.6----5.0----2.6----3.1--- ... 11.2 ->  # data channel 1
-100---105----120----96-----101--- ... 52 --->  # data channel 2

As you can see in the above schema, there are two data channels (although, ideally, the format should support an arbitrary number of channels) whose values evolve over time. Note that the time channel need not be an actual channel, per se, but was illustrated as such simply to show that video and data channels are represented as a function of time.

At the very least I'd like to be able to populate each "index" of each channel with an int or a float. Ideally, however, I would like to be able to hold a reference to some binary data of arbitrary length.

In addition to providing the above functionality, this hypothetical codec should support some form of modern video compression. Does such a thing exist?

gnat
  • 21,442
  • 29
  • 112
  • 288
Louis Thibault
  • 2,178
  • 3
  • 16
  • 17
  • http://meta.programmers.stackexchange.com/questions/6483/why-was-my-question-closed-or-down-voted/6487#6487 – gnat Jul 30 '14 at 11:30
  • 1
    @gnat, with respect, your comment suggests you've misunderstood my question. This question relates to program design, specifically the **kind of video format that is best adapted to my problem**. The request for libraries is both secondary and incidental. As such, I would greatly appreciate it if you rescinded your downvote. – Louis Thibault Jul 30 '14 at 12:08
  • no matter how you twist it, spammy answers like "use `this library`" would be legitimate in the context of the question – gnat Jul 30 '14 at 12:11
  • 1
    @gnat, no they would not, as they would be missing the essential bit of information: "this codec does what you want". If you'd like to edit my question to remove the `bonus question` bit, please feel free to do so and I will submit to the peer-review, but this is not a reason to downvote an otherwise appropriate question. – Louis Thibault Jul 30 '14 at 12:12
  • @gnat, I've edited my question to make it clear what the real question is. I think this is a fair compromise, no? – Louis Thibault Jul 30 '14 at 12:16
  • with all due respect, I fail to see how the edit would make spammy library recommendations less legitimate. While we're at it, why don't you just wipe this part out? – gnat Jul 30 '14 at 12:18
  • @gnat, No offense taken =) The issue here is that "use library x" doesn't answer the main question which, again, is "which codecs support arbitrary data channels?". A library suggestion is relevant *if and only if* it follows an explanation of which codecs solve my problem. And again, I'd be happy to submit to the peer-review process, but that only happens if you (or someone else) submits an edit. – Louis Thibault Jul 30 '14 at 12:21
  • 1
    @gnat, screw it. I'll just remove it. No use getting in a gridlock. – Louis Thibault Jul 30 '14 at 12:29
  • missing the part you call "main" would only make an answer _bad_. Bad, but still legitimate, in the sense that one can't flag it and expect moderator to delete (mods typically abstain of post quality evaluation, and this is the right thing if you think of it, quality rating is for community members). That's the issue with resource requests, they open a door for spammy garbage answers to legitimately leak in – gnat Jul 30 '14 at 12:35
  • 1
    The SMIL markup language does precisely what you're asking, timestamped metadata for multimedia.http://en.wikipedia.org/wiki/Synchronized_Multimedia_Integration_Language – Michael Brown Apr 17 '15 at 07:35

1 Answers1

2

Subtitle files typically holds textual data, but there's no reason why you can't use it to associate arbitrary data or a reference to the data (if the actual data is too long for convenience).

Note that subtitle files are usually time based rather than frame based. If it's crucial for your app that the data are associated with precise frames, you'll need to use a subtitle format that is specified in frame intervals or adapt an existing format but replace its time intervals with frame intervals.

Some of the XML based subtitle format would be fairly easy to adapt for this.

Lie Ryan
  • 12,291
  • 1
  • 30
  • 41
  • This is a pretty good idea. In principle, one should be able to integrate these files directly into a container file of some type (e.g. .mkv), no? – Louis Thibault Jul 30 '14 at 16:51
  • @blz: yes, although a video player might try to display the embedded subtitles, which may or may not make sense depending on the data. Video formats/containers that supports embedded subtitle would also usually support multiple subtitle languages which you can use for multiple data streams. – Lie Ryan Jul 30 '14 at 16:56
  • this sounds like the way to go, then. These video files won't be played with a video player. Instead, they just need to be accessible using standard tools. Provided subtitle tracks don't impose severe constraints on the length of the binary data, this seems like a perfectly useful hack. I'm going to keep this question open a bit longer, but there's a good chance I'll accept your answer. Thanks =) – Louis Thibault Jul 30 '14 at 17:00