Re: IoT Gateways: Designed out before designed in?

stevegraves — Tue, 07 Jun 2016 10:58:08 -0000

Another justification for Gateways is distribution of processing. It is unlikely the cloud needs the highly granular data generated by endpoints, and the necessity of processing all that highly granular data would require incrementally more cloud computing resources and/or slow the processing. Rather, data should be gathered at gateways and pre-aggregated before being shipped up to the cloud. For example a temperature gauge might generate data every second, but a gateway could collapse that to 5-minute high/low/mean/median for hundreds of sensors, relieving the cloud of that processing burden and dramatically reducing cloud storage requirements.

I also doubt the contention that security "need flows through the veins of every developer involved at every level." Or that "pretty secure" is good enough.

Re: VectorWise

stevegraves — Tue, 11 Aug 2009 13:12:56 -0000

Interesting article – but why no mention of another innovation in data management that’s emerged in the past decade, the in-memory database, or IMDS, which eliminates disk I/O entirely? And not just I/O, but other sources of performance overhead you might not think of, such as cache management and the many transfers of data in a disk-based DBMS. My company, McObject, makes an IMDS, the eXtremeDB embedded database, see http://www.mcobject.com/ext.... Or Google “in memory database” for some other available products.

I've read your other blog entry on in-memory databases (http://blog.tonybain.com/to.... A key point you didn't mention there is that in-memory database systems can require a fraction of the storage space compared to on-disk DBMS, so the limit to the practical size of an in-memory database may not be as severe as you think. This goes back to the optimization goals that you touch on in the article: in-memory DBMS optimize to reduce CPU cycles and the storage space required; on-disk DBMS optimize to reduce I/O at the expense of storage space and CPU cycles. Bottom line: a 100GB on-disk database probably doesn't require 100GB of memory when that data is moved into an in-memory database.

Note here that a distinction must be made between a true in-memory database system versus a database that happens to be in memory. Due to the growing popularity of in-memory DBMS, many on-disk DBMS vendors have added so-called "in memory" capability, but only by moving the storage from disk to RAM, and so they don't use different algorithms (such as eliminating the cache) or store the data any differently (so a 100GB on-disk database will still need 100GB of storage space even if "storage" is RAM).

Disqus - Latest Comments for stevegraves

Re: IoT Gateways: Designed out before designed in?

Re: VectorWise