Disqus - Latest Comments for dluebbert

Re: Better Than Anything

David Luebbert — Mon, 01 May 2017 17:01:29 -0000

Sheet music for "Better Than Anything" can be found in the Real Book, Volume 5 (https://www.amazon.com/Real.... If you need access to Volume 5 quickly online, it's available at scribd.com (https://www.scribd.com/book... if you maintain a subscription at that site, which runs for $8.95/month.

If you've never previously subscribed to Scribd previously, the site's current policy is to give new subscribers free access to all materials on the site for the first month. They do allow you to cancel that subscription before billing begins at the start of the second month.

As far as I can tell all five volumes of the Hal Leonard versions of the Real Book are available there. (https://www.scribd.com/sear...

It's not possible to download all of the music in the Scribd versions of the RealBooks in single downloads. Scribd gives you access to the individual songs that you request to see one at a time.

As I was browsing around, I also found that Scribd also lists a number of the Brazilian music songbooks by Almir Chediak (five volumes of Bossa Nova songs, the three volumes of his Tom Jobim Songbooks, the two volumes of his Ivan Lins Songbook, the two volumes of his Djavan Songbook, the two volumes of his Chico Buarque Songbook), with each entire songbook downloadable in PDF format. Here's the list I found on the site (https://www.scribd.com/sear...

Chediak worked for decades to produce collections of the work of the most important composer/songwriters of Brazilian popular music. When I run through my mental list of important Música popular brasileira composers and compare that with Chediak's offerings, I only wish that he had been given opportunity and time to document Milton Nascimento's and Tom Ze's music.

After Chediak was murdered in 2003, the publishing for these important books fell into a shambles. It seems to be impossible to find new copies of Chediak's songbooks to buy. On Amazon, used copies of these books run in the $75 to $90 range (https://www.amazon.com/s/re....

Buying Scribd access for a month or two surely would be worth it, if it happens that you are interested in investigating Brazilian music or want to collect the Real Book scores that you have greatest interest in.

Re: Why are Seattle's bridges so high?

David Luebbert — Wed, 02 Nov 2016 02:01:01 -0000

Also, there was a zone of marine business on the north side of the ship canal that was established decades before those bridges were built. By designing the spans of the new bridges to fly above all of that, the engineers preserved the east-west flow of traffic along the canal that those waterfront activities depended upon. The Roosevelt Bridge was preserved underneath the Ship Canal bridge and the Fremont Bridge below the Aurora Bridge so that the much slower local traffic that fed the marine activity could move north-south across the canal on the those older drawbridges.

If the Aurora Bridge were built lower, it would have bisected and destroyed the Fremont neighborhood which is still a very vital and interesting part of Seattle. With the bridge built high, if you drive or walk through Fremont, you barely notice all of those vehicles that are streaming high above you.

In recent years, Google, Adobe, and Tableau Software all have built offices in Fremont to take advantage of the nice views along the Ship Canal just to the west of the Aurora Bridge.

Re: This Is For Albert

David Luebbert — Tue, 22 Mar 2016 00:23:17 -0000

You're right. I didn't do the comparison of facts that you did, and notice that this was recorded several years before Bud died. I laid down in memory this idea about the tune's origin a few years before I later learned the details of Bud's biography. I never realized I needed to revise my thinking on this point until you wrote.

By time I heard the album, Bud had passed away 10 years before, so I thought of this as a memorial piece, similar to "Lester Left Town", which was written also in the early 60's, to remember Lester Young's personality and music after Lester died.

I definitely got it wrong. The liner notes for the "Caravan" album say "It is dedicated to Bud Powell -- although standard reference works don't list it, Wayne says Art (Blakey) and other contemporaries of Powell insist that the pianist's actual first name is Albert".

The title doesn't refer to Bud in the past tense. It's "for Albert" so Wayne must have thought of it as musical gift or homage that Bud might have a chance to hear and appreciate.

I'll fix my intro above. Thanks!

There's a neat story in the Michele Mercer biography of Shorter "Footprints" that tells of Bud Powell visiting Shorter at 3AM in his hotel room a few hours after they played together at a Jazz Messengers concert in Paris. Bud didn't explain why, but he asked Wayne to play something for him on his horn. Shorter played him Bud's "Dance of the Infidels" and soloed a bit, staying close to the melody.

'After he played, Bud thanked him, stood up and walked to the door. He turned around and stared. "Are you all right?" Wayne asked. "Uh-huh, it's all right," Bud mumbled in response.

"Bud had that wild look," Wayne remembered, "but it was contained. I knew about Bellevue psychiatric hospital and something-otomy and how they tried to shock him. But when he went down the hotel stairs that night, he walked straight, with his hands in his pockets. He wasn't ripped or anything." But if Bud wasn't looking for a fix, Wayne didn't know what to make of his late night visitation. He vacillated between convincing himself it had been an apparition and puzzling over possible motives for the visit. Why him and not Lee (Morgan)? And why had Bud requested the solo concert? Wayne deposited the episode in his memory banks, though with time his interest in understanding its significance only grew. '

In the early 80's when he was cast to be part of Bernard Travenier's film "Round Midnight", which was loosely based on Powell's life in Paris in the late 50's, Shorter met Bud's daughter when she visited the set.

'She was particularly struck by Wayne's story of her father's appearance in his hotel room. "You all had played together, and Father was a guest with the band that night?" she asked him. "You know, Father didn't go up to see you to get money; he didn't sneak and hide from people to see if he could find someone to help him get drunk. Father followed music, and he went to see you for the reason that he said. To make sure that everything was all right." Wayne couldn't imagine what she meant. "For the future of music," she explained. "He heard something going on with you that night when you were playing."

So apparently "This Is For Albert" is a gesture from Wayne to Bud Powell that might have been meant to reciprocate for the honor and respect that Bud paid to Wayne by asking him to perform solo on demand in the wee hours after a show they played together.

Re: Re: How do I keep the Song moving on?

David Luebbert — Tue, 10 Nov 2015 15:32:44 -0000

There's a web service, Disqus, that I'm using to comment here, that is in charge of my profile for these discussions. If you click on my photo, you'll see my Disqus profile page.

For what you are trying to do, I think you'll want to set up a profile on Disqus, and then use your ID there to reply to posts here.

Unfortunately, Disqus still does a poor job of letting one post music materials in a discussion reply. For that reason, I have maintained the original discussion response thread mechanism that I used when I first created the site.

I'll see if I can do a better job of integrating Disqus into Discussion messages on SongTrellis.

This site is pretty much a one person project, so I have to invest my time carefully so that I do the most good with the time I have available for the site.

Re: Re: How do I keep the Song moving on?

David Luebbert — Tue, 10 Nov 2015 15:24:16 -0000

Chuck,

I’m sorry that I’ve caused confusion by speaking over your head. After I learn more about what you do, and what you are trying to do, I should be able to be of better use to you.

I’d like to take one more crack, and demo a tonematrix for you, so that you can see what they do and how powerful they are.

Once it’s been demoed I think you’ll be blown away by seeing how fast you can find new music using a tonematrix.

I’ve done a set of screencasts on YouTube that show how they work. (A screencast is a video captured from a computer screen that demonstrates how a piece of software works, with a voiceover that describes what’s happening on screen).

Here’s the URL for the first tonematrix screencast that I did

https://www.youtube.com/wat...

In that video, I invent a measure of music in about a minutes of time by clicking on squares within a constantly cycling music animation.

Using that original invented idea, I find a number of variations of that original idea, and then subtly change the rhythm of the idea until I really like how the idea lays across the beats of the measure. Then I save the idea as a MIDI sequence. Since this screencast is taking place on a modern OS X Mac, the midi file that’s saved opens immediately in Garage Band, where I try a number of different ways to reorchestrate the idea that I saved.

Why it’s worthwhile to know about the tonematrix: A tonematrix lets you make changes to a continuously looping small piece of music without interruption. It allows you to turn on a sound within a slice of time within a measure to see if a sound is a good addition to your piece, and lets you turn it off again if it doesn’t work. Because the music is always cycling, you don’t waste time selecting music and starting it repeatedly. And because the music that you are inventing is revised nearly instantaneously, you get instant feedback on what your idea sounds like as you change it.

You don’t worry what symbol is used to invoke a sound, what a sound’s name is, or what actions you must take to produce the sound on a keyboard. Instead you just think of an idea as a shape that moves up and down that has gaps within it that cause the music to pause momentarily every so often. You draw that shape into the matrix, and quickly change that a bit at a time until the matrix is playing music that you want to keep.

Tonematrixes can be saved as clickable buttons, that show a reduced image of the matrix. They are great for keeping track of music that you’ve invented. You can post these buttons in web pages so that others can launch and see how the animation for an idea works. They can change the original music and change it into something that they can use themselves.

When you click on a TM button, the matrix page launches, and the TM starts to animate as soon as its music track can be created and loaded into your browser. Frequently a Play button is shown below a Tonematrix so that you can audition its music, so that you can quickly hear what it is before you decide if you want to launch its animation.

If you like that first one, the second screencast

https://www.youtube.com/wat...

shows some serious magic at work. It shows how you can do a mood transplant on a melody idea: you make a chord or scale selection on the tonematrix page and then press a button to cause the intervals within the original melody idea to stretch or shrink to fit the new set of pitches that you’ve selected, producing a similar, but different sounding melody.

If you’ve found a melody idea that you like that you found using a tonematrix, its row pitches are automatically bound to the notes of a particular chord or scale. If you change those row pitches to use a different chord or scale, in an instant you morph the original idea into an entirely different melody that keeps the shape and timing of the original idea, but which evokes a different mood, which is caused by the different configuration of pitches used in your new chord or scale choice.

Re: my roland xv-88

David Luebbert — Tue, 10 Nov 2015 13:34:51 -0000

Chuck, sweet device. I have Roland JV-80 from a few years before that, which is probably a predecessor to your synth.

Re: Tiffany and co Black Roman Numberical Ring - $67.00 : Profes

David Luebbert — Wed, 04 Nov 2015 04:32:48 -0000

Now check this tonematrix
You will hear the same one measure idea, played through each of four different chord choices.
The thing to notice is how much melodic variety can be generated following the contours of one idea.
Given any chosen unit of melody, its possible to prepare an extremely large number of contour following variations that all sound good.

Re: Tiffany and co Black Roman Numberical Ring - $67.00 : Profes

David Luebbert — Wed, 04 Nov 2015 03:57:02 -0000

I want to give you some examples that you can listen to that illustrate these ideas, that you can do experiments with.

First, follow this link

This will take you to the first idea on a page that will allow you to audition four different musical ideas. Press the first "Play" button that appears towards the top of that page. That will perform the musical idea that is drawn out immediately above that button, as a sequence of red squares drawn in a rectangular grid.

In that notation, grid columns represent even subdivisions of time within one measure of music. The grid rows represent one pitch from a chord or scale that will accompany that measure of music.

If you mouse click within that grid, a page will launch which will perform the music specified that grid that will also animate the grid in synchrony with the music that plays.

This link will also launch that page.

If you let that page sit for several seconds, the music track for the animation should be delivered to your browser in mp3 format and the animation will play.

Be patient for the download to complete. Even the mp3 for a single measure of music can be sizable.

Just to the right of the animation grid, there are a pair of dropdown menus, one labelled "Chord root" and the other labelled "Type". Together these menus specify the root and type of the chord that is providing the pitches that are assigned to the rows of the matrix.

If you change the settings of either of these menus, then press the "Change Performance Now" button that appears several lines below the chord assignment menus, you'll cause the Tonematrix server on SongTrellis to change the pitch assignments for the matrix, and play the music that corresponds to that change i synchrony with the matrix animation that is performed.

There are definitions for 52 different chord types built into this Tonematrix service, and 12 distinct roots upon which these types of chords can be built, giving a total of 625 different chord and type combinations that might be selected.

Eventually you'll realize that nearly any choice you make is going to generate a new melody that fits whatever harmony you've selected.

Re: Tiffany and co Black Roman Numberical Ring - $67.00 : Profes

David Luebbert — Wed, 04 Nov 2015 03:48:01 -0000

Chuck, I think of a piece of music in terms of small chunks, that are included in larger chunks, that finally are sequenced one after another to create entire multi-measure sections of the piece.

When I talk about motifs, I mean a small memorable idea of a few notes that I can use repeatedly (more than once) to build larger structures within my composition.

How do you know that you might be hearing something that could be considered a motif? As you write a new melody, you'll start hearing it as a series of chunks (perhaps so small you might call them chunklets). One of these chunks, if it has a distinctive flavor or aura about it, might be a motif candidate.

Such a melodic chunk should be called a motif when it is invoked later on in a melody being composed, in one or a few later spots as a way to extend the melody.

The next higher level of organization in a melody is a phrase. You know you are hearing a phrase, when your ear identifies an area of the piece that starts in motion, continues at a certain rate for a number of moments, and then finally comes to a resting point, usually marked by a rest or a note with a prolonged duration.

Musical phrases correspond to the phrases of text that make up a written sentence. Musical phrases are sequenced one after the other, until an ending phrase is reached, where the ear tells you the sequence has come to a major resting point, thereby emulating the structure of the phrases that make up a sentence of written text that finally comes to a period.

Such completed phrase sequences are meant to fit together to fill the next larger structure in the piece, that I'll call a section.

Sections are fitted together in a pattern, similar to a rhyme scheme in a poem, that specifies how the different sections of the melody are presented to the listener.

When you speak about A B A A, A B A B, or A B A C forms, you are describing the order in which different sections are introduced to the listener in a melody.

So using this nomenclature, I'd translate your question "how do I know when the motif, should go to a A B A B or A B C D ETC.OR ABBA?"

into the question "how do I know that the pattern of sections in a piece, should follow an A B A B, A B C D, A B B A, or A B A A form or some other structure that I invent".

My answer to this query is that there is no real rule that can help you to decide this beforehand. When you compose, you provisionally decide that you are going to try to compose using one of these rhythm schemes, and then you compose the constituent sections and then try them in the different sequences that you are trying experimentally. When a structure resonates in your memory, and the pattern of repetition adequately rewards you, then you consider that repetition scheme to be a keeper.

These kinds of sequencing and repetition ideas that you are applying to the larger sections of a piece, can also be applied to the smaller phrase sequence within sections and to the pattern of motif invocation within phrases.

As you move to the smaller units of organization though, you probably don't explicitly choose a pre-selected repetition structure beforehand.

Instead as you compose, if you've used a particular motif earlier, you might wonder to yourself, if it has something to contribute to the little chunk of the piece that you are composing at that instant.

If such an impulse works out for you, you've successfully extended your piece by a little, and you hear a bit of echo that reminds you of something that happened earlier in the piece that feels good to you.

You say "I feel the ear takes me different places, but I sometimes cant find a chord to go with the motif". This tells me that your idea of how to use a motif is a bit too rigid.

Think of your motifs as musical units that can be stretched or compressed like taffy to fit a new harmonic situation. Your motifs provide a shape that can be used to derive a new set of pitch selection.

So the task is not to find a chord that lets your invocation of a chosen motif sound as an unchanged sequence, but instead you find a chord or sequence of chords that extends the harmonic sequence of your piece in a way that makes sense to your ear, and then you stretch the pitches of your motif, making minimal changes to it, mostly maintaining the original contour of the idea until you've transformed it into a closely related idea that fits the underlying chord.

When you listen to the original motif over a new chord, you could get unlucky and discover that no pitch of the original idea fits the new harmony. But we know that its extremely likely that you will find a pitch that fits the harmony better, within a distance of a minor or major second from the original pitch.

So by slightly deforming the original motif, jittering pitches up or down by small steps and listening until you've derived a related idea that seems balanced and interesting to listen to in the new setting, you've found a great new idea to add to your piece.

Re: Tiffany and co Black Roman Numberical Ring - $67.00 : Profes

David Luebbert — Sun, 01 Nov 2015 11:55:24 -0000

Chuck, great to hear from you.

I can go on about this at length, probably enough to be overwhelming for most folks.

Before I get started, could you tell me how you invented or found the motif that you'd like to expand into a larger melody?

Did you sit at the piano and write an idea down? Possibly you sat at a MIDI keyboard, and captured a phrase that you played with a MIDI sequencing program.

Knowing how you have your melody represented or recorded and knowing what composing tools you have available will help me tune advice I give you for your specific situation.

When I hear someone talk about a musical motif, I think of a small, memorable musical idea that I hear as a single musical gesture. If I'm composing with a motif, it's idea that has attractive features. I enjoy listening to it and likely wouldn't get tired of it if I listened to it repeatedly..

This is close to your motif idea too, right?

Once you have a motif in hand that you've used at the beginning of your melody, you'll want to repeat it. So you copy it and play it at some later moment in your piece.

As soon as you do this, you'll establish a rhythm using repetitions of your idea. You'll likely find that if you copy the entire measure that holds your idea and repeat it as the next measure of your piece, the motif you're using means exactly the same thing that it did the first time it was played.

If you begin the second repetition at any other starting place in your new piece's structure, on a different beat or different beat subdivision, you'll have drastically changed the rhythmic meaning of your idea, and added interesting variety to the melody that you're creating.

So if you take your motif and start it a number different places later in the piece, it will sound different to you, and likely you'll find starting points that give a rhythmic effect that you find really enjoyable. When you make this kind of discovery, you keep it until you find something better to do at that point in your piece.

What features do you remember about a motif?

If you're like most listeners, listening rhythmically you'll sense the length of the idea, the velocity at which it travels through your mind. You'll hear the variety of durations used in the idea, and the points where the flow of durations changes, and you'll hear points where your melody pauses momentarily then continues. You'll notice whether the notes of your idea fall on the beat or fall between beats.

Listening melodically, you'll notice how the melody moves up or down, its direction. You'll notice the size of leaps that your ear senses between melody notes. Likely you'll hear your melody make a series of small steps and then occasional larger leaps. You'll notice if the movements that a melody makes have a pleasant consonant sound or a more disturbing or more active dissonant sound. You'll notice what kind of sensations are used at the beginning, in the middle and at the end of the idea.

You'll sense whether the ending of the idea seems like it has ended or is somehow incomplete, whether the ending was expected or a surprise. You'll also sense whether the begging of your idea was expected or a surprise.

Listening harmonically, you'll notice how each note relates to chords that are sounding in the accompaniment, and what intervals each note makes with any other melody that's playing in you piece. It's likely that the longer duration notes in your melody will play a pitch that's consonant with the pitches that are playing in your chord accompaniment, and that some of your shorter duration notes make a more dissonant sound with the chord accompaniment. You'll notice if your melody is held across the change of chord in your piece.

Your ear will almost certainly get a big jolt when you play your idea in a new harmonic setting.

As you can see, a motif is a small bundle of memorability, that has a large number of audible features that allow you to identify it.

If you changed every feature of your motif, you'd have to say that you are using an entirely different motif. But if you vary only one or two or a few features of the motif, leaving everything else the same, you'll hear a new idea that has an extremely strong family resemblence to your original idea.

If you change the duration of a single note someplace within the motif, you'll cause later pitches to slide forward and back in time and to fall differently with respect to the beat.

That may be a pleasant change to make to your original idea. Most likely you'll want to slide the later part of the idea forward and back in time until you've found so that it lays at an interesting moment in your melody. You'll find that most others don't sound as good as the change that you thinks sounds best. You just continue to change the position of the later notes until they sound right.

You can change the pitch of a single note or of a few notes and see how that sounds. Very likely it will interest your ear to hear you original idea and then hear how you've morphed it.

Most likely the harmony will change for your melody every measure or so. If you copy your motif and plop it down so that it plays over a new chord, its likely that some pitch in the new usage will dissonantly howl in its new position. When that happens you slide those pitches up or down to pitches that fit the new harmony better. If you slide pitches around, mostly maintaining the contour of your original idea, you'll create a variation that still has a family resemblence to your original idea that you enjoy hearing in the the new harmonic setting.

To summarize, you use a motif in a piece by placing it at earlier or later moments in your piece and transforming it into different shapes that properly fit with the other rhythms of your melody, the earlier and later melodic feeling of the melody, and with the sensation provided by the melody's harmonic accompaniment.

If you are composing motifically, you probably will use a number of different motifs in your piece and invoke them in different patterns that you enjoy hearing.

There, that's enough to get you thinking.

Tell me more about what you are trying to do, and I'll try to give you new or better ideas.

Re: Little Pork Chop coming back

David Luebbert — Mon, 25 May 2015 14:21:20 -0000

Would like to use this:
https://twitter.com/davidlu

Re: RE: Love For Sale

David Luebbert — Mon, 03 Nov 2014 15:45:00 -0000

Marianne, when I was first debugging the Excerpt Service, there were postings that would cause that code to blow up for one reason or another. Because of that, I must set a switch in the description for each music posting before the "SongTrellis Excerpt Service" link is allowed to be offered on that page.

That switch was turned off for "Love For Sale". Now that the excerpt service code has been become stable, it will handle almost any sound posting that is a Type 1 MIDI file, so even when I've missed a page and left the Excerpt Service switch turned off for it, it will nearly always operate correctly now when permission is given.

I've turned that permission switch on for "Love For Sale". It seems to operate safely. You should see that the link for the Excerpt Service appears towards the bottom of that Changes page, where you expected to see it.

If you find other sound postings that are missing their Excerpt Service ink, let me know and I'll try to activate it.

In the next day or so, I'll scan the database and see if can find others that I overlooked.

Re: Tweetstorming

David Luebbert — Mon, 30 Jun 2014 12:04:38 -0000

Twitter has pre-built support for threading tweet replies that meshes very nicely with tweetstorm submissions, when each tweet in the storm is submitted as a reply to the previous tweet and when the same addressee list is used for each tweet.

When that kind of tweet entry happens, Twitter notices and groups the tweets of the submission sequence together and shows a vertical rule between the tweets to show that they belong together.

It also reverses the order of tweets in the presentation so that the tweet sequence can be read from first to last.

If the next tweet in the storm is always a reply to the previous posting, tweet n+1 is added to the tweet grouping in exactly the desired order.

Here's an example of such a Twiiter created Tweetstorm thread.

The main advantages of using this facility.

1) It's no longer necessary to include sequence numbers in the tweet stream to demonstrate how the tweets are threaded together.

You can still do the Wurtemburg Theses, ala Martin Luther, as a form of tweetstorming if you wish to facilitate reference to discussion of the points of a tweetstormed argument outside of Twitter, but for discussion of the subtweets inside of Twitter it's only necessary to reply to a subtweet of the storm.

2) The tweetstorm can be read just like any other text from top to bottom.

3) Once more than two tweets are threaded, Twitter starts to ellipsize your tweetstorm, by omitting some of the interior tweets in the sequence and instead providing links to expand the ellipsized tweets if you are interested in the presentation. Twitter also provides an expand button off to the right of the first tweet in the sequence that allows you to expand the entire sequence.

4) Sequence numbers need be used only when they help the argument in the tweetstorm. The numbers won't have to be used as the necessary glue that allows the tweetstorm to be recognized as a unit of text.

These collapse/hide features are great because they constitute an answer to the objection that the storm swamps the recipient in a blizzard of tweets that can take up a lot of real estate in their tweet stream.

There is difficulty in the use of this variety of tweetstorm entry, that I suspect Little Pork Chop could easily heal:

It's still easy to have your tweetstorm thread be broken through no fault of your own, if a reader decides to reply to one of the early tweets in the storm thread before you've finished typing the last tweet in the sequence.

In the example thread that I pointed to above, you can see my tweetstorm thread was broken when Glenn Hall decided to reply to me while I was still submitting the later tweets of my storm.

Once this breakage happens, it becomes necessary to enter a sequence number in your next tweet to indicate how the reader needs to jump between the tweetstorm islands that have been formed.

It would be cool if Little Pork Chop was extended to create a tweetstorm thread using the 'N+1st subtweet replies to the Nth tweet' style. Because it would submit the tweets of the storm all in one blast, there would be almost no chance for a respondent to reply to a middle tweet before Twitter established the entire thread for the storm.

It would also make it unnecessary to add sequence number the subtweets of a storm if the writer doesn't wish to write in numbered thesis form. That would reduce the visual clutter that tweetstorming creates in recipient tweet streams.

If Twitter could wait a second or three to allow the next tweet in a storm to hit using this threaded entry style, they would be able to present the entire tweetstorm as a package, which would really cut down on the clutter in recipient's tweet streams, which answers a major complaint about tweetstorming.

Re: Microsoft Word for Windows Version 1.1a Source Code

David Luebbert — Fri, 09 May 2014 16:30:34 -0000

I just documented how Word recorded property data:

Making Rich Text Rich: how Microsoft Word records a doc's property data

At the beginning, I show how the Computer Science idea of a property, which probably seems arcane when one first encounters it, likely derives from our culture's idea of personal property and the way chemical properties are described for science. Then I describe what 'rich text' means since I couldn't find a published definition that pleased me.

I recount Word team's aspirations to provide unbounded rich text editing for its customers, and how the competition to do editable rich text page views, which Mac Word and Full Write Pro engaged in between 1987 and 1989, hinged on the those app's relative abilities to record rich property sets without obvious bounds.

Finally I document the property representations Word uses to store prop info: one that was recorded in document and scratch files to handle full saved text recorded in Word document files and newly typed text recorded in scratch files, and a different representation for the property edits that had been performed since the last full save.

Re: Making Rich Text rich: Word's property encoding scheme

David Luebbert — Fri, 09 May 2014 15:03:35 -0000

How Word's property storage structures mate with a document's piece table to record a document's text and keep track of its properties:

Character Position coordinates (CPs) in Word are the indexes that identify every character that's recorded in a Word doc.

How a character is fetched for a particular CP

Word's FetchCp() routine does this job

We binary search the piece table index with a character's CP to retrieve the piece table index in order to locate the piece descriptor that describes the range of text stored in a disk file that contains the character we're searching for.

The PCD (Word's Piece Desscriptor) records the file number of the file containing the character and its file character offset (FC) measured from the beginning of the file. If ipcd is the piece table index for the piece containing the queried character, we can subtract the ipcd'th CP of the piece table index from the query CP to produce the offset of the character in its piece. When we add this offset to the beginning FC of the piece, we know the precise FC of the desired character, which we can fetch directly.

Every file that points to a Word doc file, records a CHP bin table that allows one to fetch CHPs (character properties) from that saved document, and a PAP bin table to allow one to fetch its PAPs (paragraph props).

How the paragraph bounds and PAP are fetched for the paragraph that contains a particular CP

This task is performed by Word's CachePara() routine.

To fetch the PAP (paragraph properties) and bounds for a paragraph that contains a particular document CP, we first determine the file number of the file that contains that character and calculate the fc offset of the queried CP within that file just as if we were doing a character fetch.

Then we take that fc offset value, and binary search the FC index of the document bin table for PAPs. That produces a ibte (an index to a bin table entry) for the page that contains the FKP page corresponding to the query CP that may contain a PAP recording for that character's paragraph bounds, if they are located within the piece.

Knowing how many characters precede the queried cp in its piece, we examine the fc ranges recorded in the FKP and determine if the paragraph end of the preceding paragraph is recorded within the tail of the piece.

If its not, we search linearly through pieces moving towards the beginning of the piece table, and look up fc locations for paragraph marks in each piece's file (query the file's bin table, access the FKP page the bin table entry points to, search for any fc ranges that intersect the piece), until we find the nearest in-piece paragraph mark. We calculate the corresponding cp and we've located the cpFirst of the paragraph containing the query cp.

Now, knowing also how many characters follow the one with the queried cp in its piece, we examine the fc ranges recorded in the FKP and determine if a paragraph end is recorded within the tail of the piece.

If that's not true, we search linearly through pieces towards the end of the document, and look up fc locations for paragraph marks in each piece's file (query the file's bin table, access the FKP page the bin table entry points to, search for any fc ranges that intersect the piece), until we find the first in-piece paragraph mark.

Finally we've found the piece that contains the first paragraph mark after the query CP and know that character's cpLim and fcLim coordinates.

Then we take the fc index of the paragraph mark that we've located, and access the offset of the PAPX recorded in the FKP for that fc. We add that offset to PAPX to the beginning address of the FKP page, to locate the beginning address of the PAPX for the paragraph mark.

We fetch the stc (style code) for the paragraph from the PAPX, and retrieve the PAP setting for that style and copy that into a buffer. We apply the grpprl that is stored in the PAPX if the grpprl length is non-zero. That changes the PAP buffer to the PAP settings recorded the last time the document was full-saved.

Then we access the prm (property modifier) of the piece accesses the PRC which that prm points to tanned apply any paragraph sprms that have accumulated there since the last full save. This operation produces the final state of that paragraph's PAP as the user set it. Tak complete.

Re: Making Rich Text rich: Word's property encoding scheme

David Luebbert — Fri, 09 May 2014 14:24:13 -0000

I've already written an article to describe how Microsoft Word uses piece tables to allow unbounded and fast editing of a document's text and fast saving of that text into the document's representation in a stored file:
What's been wrought using the Piece Table?".

Taken together with the document's property encodings, which are described here, and it's reference tables that record constantly renumbered an readjusted structures like footnotes, annotations, bookmarks, links, etc, this constitutes nearly all of the nooks and crannies in which document information is recorded in Word.

Re: What's been wrought using the Piece Table?

David Luebbert — Tue, 15 Apr 2014 12:30:01 -0000

I agree with your view that piece tables can be thought of as 'ropes adapted to run in limited space'.

I agree that in theory we could have done a rope implementation of the Word PLC (plex of character position) structures that we used to carry piece table descriptors and many other and it perhaps it would have run in a minimally acceptable fashion in that environment.

I believe that the rope implementation, has excellent properties when constraints of memory, code size, processor performance and debugging support are relaxed, possibly the best known.

My opinion is that if we would have used that tree implementation, which seems to be a defining feature of the data object called a rope, as the base data structure for Word, we would have reduced Word's fitness in its marketplace against its competitors.

In saying this, I am evaluating its likely performance against Word's implementation, starting the comparison in 1981, the first year of Word development, and continuing the evaluation forward into the mid-90s, around the time the rope implementation achieved traction in academia.

I know that you realize that the whole word processing battle was pretty much over by then, just a matter of stamping out small brush fires, and picking up misfired ordinance. The performance and flexibility of the piece table implementation and the ability to fast save that it enabled, helped win the battle.

There are many other parts of the technology that contributed to the win. This is the first I've had the chance to document. As time allows, I intend to write about other parts of the Word internals that were also important to its success.

I will try to research, and describe the benefits and excellencies of contending technologies as I document the historical development of Microsoft's word processing technology, as I write these articles.

You're basis for argument is the theoretical excellence of the rope structure. I grant you that.

My bases for argument are the historical contingencies, constraints and accidents that allowed Word to eventually succeed in its marketplace against many challengers. I'm writing for CHM here, an organization that is interested in documenting and understanding those kinds of considerations.

If we were to recommend to today's software practitioners what piece table implementation they should use when they have a need to represent edits within a string of characters of unbounded size, in a 32-bit or 64-bit address space, and they intend to collect hundreds of thousands of edits and upwards in that table before reorganizing string order so that it's again laid out in ascending order, the best recommendation would be to do a rope implementation.

If it all possible, you would want to use an already existing Rope object that has had a lot of use, that has a great implementation of character position updating, especially if you intend to keep parallel ropes in sync during edits, a kind of thing that's frequently needed by these kinds of applications.

If for some reason, Microsoft were to package some parts of Word, into a rich text editing object, you could invoke this with little concern, for most human scaled text production tasks, like writing a book, script, etc.

If Microsoft still uses the historical edit code revealed in the Win Word 1.1 publication (for all I know, they may have converted to rope usage by this time), that object would not be the best choice for representing the entire genome of an organism, if for instance, you intended to run a process that replaced all base pair-triplets in the codon with an abbreviation showing the amino acid corresponding to a triplet, or run an automatic process that attached gene description annotations to the correct coordinates within the genome string.

For those kinds of operations, a rope implementation would be what you would want to use.

It might interest readers to note that Charles Simonyi's interests since the time of his last work on Word have been concerned with Intentional Programming, the idea that a software developer should be able to, as much as possible, declare his intentions describing what a software object should do, and have that system write, and then later revise, the code for large portions of a complex object.

Already 15 years ago, one of his intentional systems, could parse code written in COBOL, C, or other historically used languages, and re-express those texts in working LISP or other imaginable language variants.

Even now, once a sizable project has been written on top of particular data structure implementation using one of the historically important languages, it can be extraordinarily expensive to change the internal implementation so that it works in all respects as it did before the change.

There are reports that his recent implementations are robust enough, that, for instance, they could be used to parse parse projects of Word's size and apply descriptions of a particular software object representation and reliably rewrite the code to use another representation without introducing errors in the process.

That could be a task like converting Word internals to use ropes rather than its historical dynamic array implementation.

If the system were properly instructed, and a developer described the intention to edit large unbounded strings,that system might recommend using a piece table implementation, and give you those choice of which base representation to use.

Which ever you chose, it would write the code that performed that constellation of operations and performance profile, and give you the opportunity to alter that implementation after initial creation.

Re: Old-time laptops

David Luebbert — Tue, 15 Apr 2014 10:41:49 -0000

If I remember correctly, the Model 100 was the last Microsoft development project that shipped with code that Bill Gates wrote. Wikipedia confirms the recollection.

Re: What's been wrought using the Piece Table?

David Luebbert — Sun, 13 Apr 2014 15:10:59 -0000

Let's see: MacWord 1.0 ran all of its Window, menus, dialogs plus document structures, associated piece tables, and property change data in a 28K heap allocation.

I believe Mac Word 3.0's heap wasn't allowed to grow to its 64K maximum size. 44K sticks in memory.

a) We did not have the luxury of a reserving special heaps for objects of particular size. What could not be used by one class of data had to be used by others.

b) Divide 28,000 by 12 (10 byte piece description plus heap block overhead). Result is 2333 assuming no other use other than piece table. That's the largest number of edits we could represent with one piece table if the heap was used for no other purpose.

Now add in 2 bytes apiece for each tree node, plus a two byte finger table entry overhead for the heap allocation, 16 byte overhead - 1755.

Remember we did want to allow the user to have several windows open and still edit, allowing search replace to happen for at least one of them.

Divide that best case by 8 or 16 to derive a realistic figure.

Those node slabs would work for a quick open, but you'd still have to chase down new nodes that were added and reorganize at save time (remember our need for speed). And it would add to code complexity to track down and reuse tree nodes that were edited out.

We were gasping for heap space, execution speed, and reduced code size.

Those node icebergs would also have taken away space that we needed for other purposes.

I hope I manage to convey the constraints under which we operated. We were constrained, mon!

Your recommendations might perhaps have become effective and begun to pay off sometime during the early 90s.

Piece tables can be implemented using several different data structure ideas: linked lists, dynamic tables, ropes (a binary tree adaptation), or Red-Black trees.

Using the term, I'm trying to describe Word's implementation, which was adapted to its environment . I'll gladly share with others.

Re: What's been wrought using the Piece Table?

David Luebbert — Sun, 13 Apr 2014 14:14:53 -0000

Look, my audience as I wrote this was meant to be the visitors of the Computer History Museum. Most readers will find their way to this article by way of link from that announcement page. That link was established yesterday.

I write knowing that thousands who are untutored in Comp Sci may read this. You're arguing in the midst of an active museum exhibit. You'll have to accept those constraints because of the forum you've chosen.

I reiterate that the per node memory allocations and link overhead of a piece table realized by a tree could not have been used by Word in the early 80's, and probably could not have been used profitably until the mid-90s.

For my audience, I'm trying to explain why Word came to succeed in its market when it did, and explain details of execution that were significant in its eventual domination.

During this last, I was trying to explain the tradeoffs they had to make to succeed using an implementation that still had a performance problem.

From today's vantage, with memory resources effectively infinite, you can claim "there doesn't seem to be any significant difference". At that time, working under those constraints, the rope as described in 1995 couldn't have worked.

I'm not claiming unparalleled innovation, but instead a solution extremely well-adapted to its constraints, early for its time that allowed the benefits of the idea to be experienced by paying customers a decade or more early.

I am trying to show how a fairly complicated software object worked in its constituent parts and explain its performance and success in the marketplace. There aren't many available for the public to easily study.

Regarding Xerox PARC, Simonyi and Brodie left in 1981. One of the Bravo versions implemented piece tables on top of linked lists.

Charles told me during the time that the first version Mac ran around 1 MIPS, shaking his head a little, "that machine that runs SmallTalk, it clocks at 44 MIPS".

He knew that we'd be able to use those kind of resources after some number of years, but it was still fairly far off.

Re: What's been wrought using the Piece Table?

David Luebbert — Sun, 13 Apr 2014 11:09:04 -0000

I'm sorry to have caused irritation.

I don't assume that ropes nodes needed to be the carriers of the text for pieces, but in the run of "this is nothing but a rope" opinion that showed itself in the discussion of this article that appeared in the r/programming section of Reddit, participants were already settling on the idea that Word's piece tables somehow carried the text.

At first, I had assumed that rope structures were best used to record pointers into text. But seeing the misconceptions that were already being formed that piece table nodes were used as a chunky text buffer, most likely because of slightly ambiguous language in the WIkipedia Rope reference and accompanying diagrams, and because of the growing conviction that Word's implementation was nothing but a rope, I felt that it would be good to construct a prophylactic against that thinking.

WIth this addition, I was more interested in clearing this strangler vine off of the public's forming idea of the Word implementation, rather than making sure that the rope idea was untangled from parasitic foliage also.

You've done that here, and I thank you.

It is upsetting to believe that the literature for a particular area of computer science has been sorted out, long settled, and that the best solutions for an area are known, and then to have a long maintained trade secret be uncloaked, which shows that a related but better solution had been invented to solve an extremely important commercial problem decades before.

Richard Brodie's and Charles Simonyi's search for the best piece table implementation for a word processor, which would run efficiently in an extremely small address space, did not send them towards the rope solution, which was finally published 15 years after their work.

The number of piece nodes that would have to be created to represent a document during a large search/replace when thousands of nodes would be created, would have led to a judgement that this kind of implementation would not fly for their application.

They did have a glimmer of an idea that fast save would be feasible using the dynamic array implementation that they settled on. The writing and reading of piece nodes after a big replace would have been horrendous (thousands of I/O operations when one or two sufficed for Word's case), if a tree based implementation had been chosen.

Using trees would have made it impossible to bring home performance gains against competitors that were important to Word's acceptance in its market.

They did make a performance trade/off when they settled on the 'two parallel dynamic arrays kept in 1-to-1 correspondence stored within a single heap block' implementation.

Insertion and deletion of pieces became O(N**2) if you have to use dynamic arrays. This n-squaredness meant that every time the number of piece table entries grew by a factor of two, the work necessary to put the piece table back in shape after an edit or delete grew by a factor of 4.

In the early 80's, running on pcode interpreted code, this extra work started to interfere with the user's expectation of performance, once a piece table had grown to represent a document that had endured a hundred edits, whenever an insert or delete operation was allowed to occur

A tree implementation would have been able to do the updates for insert/delete in O(log*N). If this benefit could have been reaped using dynamic tables, no further work would have been necessary.

Because they accepted this penalty, in favor of the dynamic table's advantages, they had to do further invention to remove some of the sting for having done so.

First, on insertion or deletion, a single piece representing a 32 character in-memory buffer was inserted into the table when a user selected a location in their document and began to type or delete there.

That meant that the large update piece table cost would be paid only when the buffer filled after 32 character key strokes or 32 backspaces had been pressed.

They also arranged that if the user continued to edit at the back edge of the newly added piece after the buffer filled, or at the back boundary of the piece that was representing deletions because of a sequence of backspaces, they could simply adjust the boundary of that last added piece (or last modified piece for deletion) to accommodate the users newest editing operations, thereby converting the next buffer flush from an n-squared update to an order 1 operation (ie. 1 unit of work instead of the hundreds, thousands or millions of times more work that could have been required because the piece table had gotten large).

For Mac Word 4, we had invented a way to do lazy adjustments, of the PLC (plex of character position tables) that carried the piece table and the close to 15 other parallel tables, which needed to be updated whenever a document edit took place,

That optimization frequently transformed an operation that could be O(N**2) on the current size of all of those tables, into an O(N) operation in many cases.

If all of these measures were finally overtaken by the costs incurred by piece table growth, we took comfort in the notion that the user could reclaim their editing speed, after we had delayed the necessity for as long as possible, by doing a full save of their document.

>As for never having heard of ropes, with all due respect, that says more about you than it does about ropes.

I haven't worked on the Word sources for 17 years since I've left Microsoft. In my defense, I haven't kept track of Comp Sci literature on text edit problems since then.

Up to that time, the tools that we built had been shown to be adequate for their task and Moore's Law was allowing them to perform ever better.

Re: What's been wrought using the Piece Table?

David Luebbert — Fri, 11 Apr 2014 17:07:01 -0000

Also, looking at the Wikipedia discussion of ropes, it appears that rope nodes could also be used to store the text of the text stream that was being represented.

In Word, the text for a document was left on disk, and was paged into a buffer from disk on demand via a fetch routine. The piece descriptors of a piece table provided the file number and beginning file character position that was used for this fetch process.

Piece tables, and the array of character positions that was stored as the first subarray of a PLC, were consulted to calculate the beginning coordinates and extent of recorded text, not to store the text itself.

Similarly, already saved character and paragraph properties were packed into disk pages, and not consulted until a format or display operation queried them to discover the saved settings for a paragraph or run of text.

If a user never visibly scrolled past a particular range of characters on disk during an editing session, they were left untouched on disk.

The entire text of a Word file and the recorded properties for that text only flowed through memory in small buffer sized chunks during a full save operation.

During a fast save, copied or newly entered text and associated property info was written to unused parts of the fast saved file on disk, but Word never bothered to glance at text or properties that were already safely recorded within the file for this kind of operation.

Re: What's been wrought using the Piece Table?

David Luebbert — Fri, 11 Apr 2014 15:11:30 -0000

Thanks for the reference, Dag-Erling.

One of my intentions in writing this was to bring attention to the fortuity of J.S. Moore and Simonyi being at PARC in 1974, where piece tables were noticed to have beneficial properties, if your intention was to build a word processor. That was one of the first of a train of innovations that threaded through development of Bravo, Bravo X for Xerox and then Word for Microsoft.

The fact that Word's implementation, which was built on tables stored within dynamic arrays rather than upon the binary tree structure of ropes, was actually key to Mac Word and Win Word's ability to support editing while storing a document's editing state in a miniscule heap, where heap pointers were implemented as 16 bit quantities.

I won't pretend that we paid attention to the invention of the rope structure. I actually heard about it for the first time yesterday during a discussion on r/programming on Reddit yesterday. Our struggle to make a piece table implementation perform was already joined by that time.

If pieces we're stored in the nodes of a binary tree, the allocations for the individual piece nodes would have eaten our heap alive. A search/replace operation that caused several thousand replacements of the search string, would have exhausted Mac Word's 1.0's 28K heap reservation which was used to store all of the considerable number of dynamic structures that the app operated upon.

Word's internal heap grew to its full 64K maximum size for Mac Word 4.0.

For Win Word 1.0, PLCs in the internal heap (the generic data structure that carried a piece table payload, and those of many other key document structure) finally were pointing via huge pointer out to data blocks that were managed in the Window's heap.

Implementing the Word piece table on a tree, would have foreclosed on the idea of doing fast save, which became possible to do for Mac Word 3.0. The fact that most of Word's tables were implemented as a PLC structure stored in one block of memory (or two blocks for Win Word), meant that with one or two disk reads, the entire structure could be created in the heap and instantly be ready for editing operations and user type-in to occur.

If we had used a rope implementation, we would have been required to do thousands of heap allocations to create an in-memory rope when a highly fragmented document was opened, and would have needed to perform thousands of write ops to flush that structure into a new Word formatted file to save such a document. We could have saved, but it wouldn't have been fast.

Word, even when it was gasping for heap resources or computing cycles, could still open a document like lightning, and similarly could save like lightning when doing a fast save.

This was a big deal in an era when competitors were sometimes taking minutes to save extraordinarily large docs and seconds to open them.

You might be interested to review the reddit discussion that I discovered and joined at a rather late moment:

http://www.reddit.com/r/pro...

This tweet series points to my posts within that discussion tree:

https://twitter.com/davidlu...

Re: What's been wrought using the Piece Table?

David Luebbert — Mon, 07 Apr 2014 20:55:15 -0000

For one of my last projects for Microsoft, which shipped in Win Word 97
and Mac Word 98, we finally could keep track of an unbounded number of
text and formatting changes and store that inside of a Word document, an
adaptation of Word's undo feature that would have been hard to make perform
if Word wasn't a piece table editor and if it lacked the ability to fast save.

With that available, we could allow folks to
do infinite undo and also built document versioning on that capability.
Starting with a selected version, by selecting the entirety of the
Undo stack, one could run Undo all the way back to the document when it
was empty at its creation and than run redo all the way forward an
operation at a time, to watch every change come into the document, until
one arrived at the final version of the document when the redo stack
had been emptied.

Undo noticed when text was typed adjacently
and consolidated its recording so you watched text come into the
document phrase by phrase, not character by character.

This ability was lost when they finally made .docx the default file saving
format for Word (I think in Word 2004). I would guess that they thought
it was to difficult to justify bringing the Undo state forward to pour
that into a published and standards approved word processing format.

I think it would have been possible to do, if the will had been there.

I also attempted, but failed during that project, to enable document
check out/check-in for independent edit and merge of changed text,
noticing and keeping track of conflicts between the various versions
that were recorded in their accumulated stored Undo state.

That was a bridge too far. I couldn't get the change merging methods to
work correctly before we reached the shipping deadline for Word 97.

Apparently the team lost the ability to bring this feature home for the 2000
release, after I left.

Re: Tunetext lore

David Luebbert — Wed, 21 Aug 2013 19:30:00 -0000

Xastram, if you have questions about any of this, post here.

My goal with this Tunetext format is to expand its expressiveness until we can transcribe any music expressible in standard notation in the form of a URL that can be shared and used to generate a printed score for that music, an audible performance of it, and various kinds of animations that can be helpful in music study and memorization.

Also, want to include operators in the format that allow one to easily compose just by typing operators into a web page form.