Creativity in Software Development

Software development is a form of creativity.

I’m covering two subjects here

  • easiness and
  • creativity

When you are creating something, easiness is important

There is a certain threshold of work that takes an activity from supporting creativity to blocking it. This is different for every person. Those people who are highly natively creative may have a lower native threshold for work. On the other hand those who are creative have an incentive to learn how to do work in order to remain creative so their threshold will increase through learning. Still, make things to hard, and creativity goes away.
This is why easiness of software development and software development creativity go hand in hand.

The importance of easiness in building This Stack

To build the entire stack, each level needs to be quick and easy. Too much digging into the details diverts you from getting an entire application stack done and drags you off on a tangent. This is not to say that the details are not critical, in fact careful attention to detail can prevent you from debugging which is a detour/diversion

Details vs Creativity

When I’m talking about getting diverted into details here are the most common examples:

  • too fine a specification on UI is an example
  • troubleshooting and debugging get in the way of the creativity and the easiness
  • poorly designed/implemented/expressed systems tend to take very long to build and require lots of unnecessary detail work
  • too many technologies per layer

Trillions of bits have been written on how to build software easily and well.

layers and levels are different.

Creativity is a measure, not an either or

Creativity is an analog measure rather than a binary condition. What I mean by this is that there are various levels of creativity. This can best be described by an example.

If I were to create a novel, the lowest level of creativity might be
words. I can create words all day long by typing them. They might just end up being word salad, but they still get created as I type them. Perhaps a little higher level of creativity might be sentences. Putting the words together into coherent sentences – descriptions, dialog etc. is a higher level of creativity. As I type them they get created. They might not create a coherent story together, but they are still  created. A higher level of creativity would perhaps be a story. By putting sentences together that cohere into a story, I create a story. The story would have plot and tone, and theme, and characters etc. This is a higher level of creativity. The story might however be very hackneyed and derivative and boring. An even higher level of creativity would be to create a story that moves people and teaches people, puts sentences together with clever sentence forms and that most of all is a story that has pretty much never been told before. This is a very high level of creativity. The bottom line is this – that creativity exists at various levels, and this is true also in software development.

Creativity in Software Development

You create when you write a line of code. You create when you write a program similar to many that have been written. You create when you write a program that has an excellent architecture and that has been written before. The goal of ‘this stack’ is to be able to write programs that have never been written before, with an excellent architecture that allows more things to be built on top of them, and can be built out by multiple people and companies until they reach the level of understandable and computationally creative artificial intelligence that is not evil and that can work effectively with humans.

 

 

Einstein at the Patent Office

Early in Albert Einstein’s career he worked at the Patent Office in Switzerland. I have collected various snippets from around the web relating to Einstein’s relationship with the patent office.

Einstein did some of his work on the job.

“Einstein could complete his tasks so quickly and so well that he had ample spare time in which to pursue his scientific work, and was even granted a raise of 400 francs soon after being hired.”

“When he did have a few free moments during the day, he would scribble on sheets he kept in one drawer of his desk—which he jokingly called his department of theoretical physics. But Haller kept a strict eye on him, and the drawer stayed closed most of the time”

“Luckily, the young genius never gave up on his passion for physics. Whenever he had a spare moment during the workday, he would jot down notes and hide them in a drawer that he jokingly called his department of theoretical physics.”

Einstein had a circle of friends outside of work.

“While living in Bern, Einstein met regularly with a close circle of friends who shared his interests in physics and philosophy. These individuals included the Romanian student Maurice Solovine, his old friend Conrad Habicht, an electrical engineer Lucien Chavan, and Einstein’s closest friend from the Polytechnic, Michele Angelo Besso. These men met late into the night to discuss their intellectual interests and referred to themselves as the Olympia Academy.”

“Nevertheless, the friendships he had made with his former fellow students continued beyond their studies, as is reflected in his correspondence with Jakob Ehrat, or rather Ehrat’s mother”

Einstein’s co-workers helped sharpen his abilities.

“In June 1902, Einstein received the letter he’d been impatiently waiting for: a positive answer regarding his application to be a technical assistant – level III at the federal patent office in Bern. One month later he was examining inventions applications for patentability at his famous lectern in room 86, on the third floor of the building on the corner of the Speichergasse and the Genfergasse. The director at the time, Friedrich Haller, was a strict boss. However, Einstein appreciated his superior’s tough-but-benevolent and logical-but-uncompromising character which seemed to stimulate Einstein’s natural critical tendency.”

“What did the patent office job mean to Einstein? Three things:
<snip>
– second, “He (Haller) taught me to express myself correctly”, and”

Einstein’s job stability helped him focus on his work.

“The patent office job — Einstein referred to it, tongue-in-cheek, as his «cobbler’s trade» — turned out to be stroke of good fortune because it was excellently paid (3,500 Swiss francs per year) and was undemanding for his nimble intelligence. He spoke of the patent office as «a worldly cloister where he hatched his most beautiful ideas». With his courteousness and modesty and his humorous approach to life, Einstein was very well-liked. On April 1, 1906 he was promoted to technical assistant-level II. He managed his time exactly: eight hours of work, eight hours of «allotria» (miscellaneous) and scientific work and eight hours of sleep (which he often used instead for writing his manuscripts). ”

“What did the patent office job mean to Einstein? Three things:
– One – “Besides eight hours of work … eight hours of idleness plus a whole Sunday”, <snip> – third, what Einstein mentioned on his seventieth birthday, “It gave me the opportunity to think abut Physics.”

“Einstein’s job in the patent office was a blessing. After years of financial instability and depending on his father for an income, he was finally able to marry Mileva and begin to raise a family in Bern. The relative monotony of the patent office, with its clearly defined tasks and lack of distractions, seemed to be an ideal setting for Einstein to think things through. His assigned work took only a few hours to complete each day, leaving him time to focus on his puzzles. Sitting at his small wooden desk with only a few books and the papers from his “theoretical physics department,” he would perform experiments in his head. In these thought experiments (gedankenexperimenten as he called them in German) he would imagine situations and constructions in which he could explore physical laws to find out what they might do to the real world. In the absence of a real lab, he would play out carefully crafted games in his head, enacting events that he would scrutinize in detail. With the results of these experiments, Einstein knew just enough mathematics to be able to put his ideas to paper, creating exquisitely crafted jewels that would ultimately change the direction of physics.”

Einstein had trouble finding a job in his field.

“While the other graduates from his year took up their posts at the Polytechnic Institute, Einstein tried unsuccessfully to gain a position as an assistant lecturer at various universities in Switzerland, Germany, the Netherlands and Italy”

Einstein got his job through a friend.

“Grossman’s help was necessary not so much because Einstein’s final university grades were unusually low—through cramming with the ever-useful Grossman’s notes, Einstein had just managed to reach a 4.91 average out of a possible 6, which was almost average—but because one professor, furious at Einstein for telling jokes and cutting classes, had spitefully written unacceptable references. Teachers over the years had been irritated by his lack of obedience, most notably Einstein’s high school Greek grammar teacher, Joseph Degenhart, the one who has achieved immortality in the history books through insisting that “nothing will ever become of you.” ”

Einstein’s situation allowed his ideas to percolate together, coalesce, and gel into mature ideas.

“The year 1905 became an “annus mirabilis” for Einstein. Altogether he wrote five significant works on three areas of application, namely on the reality and size of the atom, about photons and about the theory of special relativity. Four of these works are mentioned by him in a letter (pdf, 535 kB) to Conrad Habicht. The brilliant concept of special relativity occurred to Einstein in May 1905 in a discussion with Michele Besso in Berne. During the following six weeks he wrote his essay, which appeared three months later in the “Annalen der Physik” under the inconspicuous title of “Zur Elektrodynamik bewegter Körper” and which would revolutionise physics as well as the established ideas about space and time.”

Einstein’s job kept him from getting sucked into current fashions in physics theory.

“Even the hours he had to keep at the patent office worked against him. By the time he got off for the day, the one science library in Bern was usually closed. How would he have a chance if he couldn’t even stay up to date with the latest findings? <snip> Einstein was slipping behind, measurably, compared to the friends he’d made at the university. He talked with his wife about quitting Bern and trying to find a job teaching high school. But even that wasn’t any guarantee: he had tried it before, only four years earlier, but never managed to get a permanent post.”

Einstein’s job helped keep him from going off the academic eccentric ‘deep end’.

“He managed to get a few physics articles published in his spare time, but they didn’t make much of an impression. He was reportedly always aiming for grand linkages that he couldn’t quite prove.”

“Einstein in the Bern patent office. “A practical profession is a salvation for a man of my type; an academic career compels a young man to scientific production, and only strong characters can resist the temptation of superficial analysis.””

“What did the patent office job mean to Einstein? Three things: <snip>
– Moreover, a practical profession is a salvation for a man of my type; an academic career compels a young man to scientific production, and only strong characters can resist the temptation of superficial analysis”.”

Einstein’s native intelligence and dedication helped him do well on the job.

“Haller then wrote, “Einstein had continued to familiarize himself with the work, so that he handles very difficult patent applications with the greatest success and is one of the most valued experts in the office.”

“His employers at the patent office were pleased with Einstein’s work and promoted him to Expert II Class, yet they remained oblivious to his growing reputation.”

Einstein’s real work was not appreciated by his boss but was appreciated by a few important others.

“Moral of the story? If you have a game-changing idea, don’t expect to get “excellent” rating from your boss, at least not in the next appraisal cycle.”

“Even by the turn of the decade (1910) there were only a handful of people in the world who had understood the impact of relativity.”

“Einstein’s idea was appreciated by
Max Plank, Lorentz and a few others. You need to find who your “Plank” is.”

“Unfortunately, Einstein’s 1905 achievements were not immediately recognized as the work of a genius. He kept his day job in the patent office for the next few years (though he did get a promotion in 1906). In 1908, Albert Einstein finally moved into academia to focus full-time on an extraordinary career as a physicist and undisputed genius.”

“Einstein was still working on a daily quota of patents in 1907 when the German physicist Johannes Stark commissioned Einstein to write his review “On the Relativity Principle and the Conclusions Drawn From It.” He was given two months to write it, and in those two months Einstein realized that his principle of relativity was incomplete. It would need a thorough overhaul if it was to be truly general.”

Einstein got to appreciate the good qualities of people outside his field.

“I love this Einstein quote: “There’s a Genius in all of us.” “

User Interface Layouts and the Ontologies Level

UI layout is a combination of 2D, graphs and hierarchies and is best handled at the ontologies level. UI layout includes both hierarchy relationships, graph relationships and 2D positioning. Each level has a native organizational paradigm. For example the RDB level has relations. The object oriented level has hierarchies of classes, and the Endemes level has lists (and perhaps more, this is still being researched). The knowledge representation /  ontologies level focuses on handling all kinds of relationships. Its native organization paradigm is probably the graph, but I see it as being the managing level for all other relationships.

For example, to efficiently, productively and beneficially do dynamic SQL generation in the OO layer, you need to have information about what each table, column, and relationship are and what these relationships all mean. This means one of the information levels. The endemes level does not seem to tend in this direction, so I proposed that these relationships be managed at the ontologies level. As ontologies are all about relationships, it makes sense that handling various kinds of relationships would dwell best at that level. Not just graphs.

Once you start using the ontologies level for UI management, then you can start including arrows in your UI’s much more frequently. In this way diagrams go from being very difficult to display and represent to manageably easy. Much of the time a diagram is the best way to represent some information. using the ontologies level for UI allows that.

Handling the way various components in a UI relate to each other is a form of information. This information is usually hard coded in a user interface layout and code. This means that users that want to see things differently can’t. This means that reorganizing a UI layout is done in code. The ontologies layer can move this into data and allow these sorts of changes to be handled by data.

Conclusion

So the best level for handling the complexities of the hybrid hierarchy and graph nature of user interfaces would best be handled at the level that handles relationships best – the ontologies level.

Note

Here are some common organization paradigms and their native levels:

  • graph [ontologies level] – A -> B [is a]
  • subclass [ontologies level] – A .. B [inherits from]  information ‘subclassing’ through information parameters and implementation
  • hierarchy [object oriented level] – A: -B  [has a]          information ‘grouping’ through some generic structure
  • list [endemes level] – A – B  [after a]        lists orderable by information
  • Relations [RDB level] – A -< B [relates to a]   table/column is addressable and characterizable, and business-rule-able

 

L.I.Q.U.I.D. as an Alternative to S.O.L.I.D.

Object orientation has a design paradigm suite called SOLID. I propose ‘LIQUID’, a design paradigm suite for information orientation.

  • L. Levels: RDB,data,info,ontologies
  • I. Indirection of fields
  • Q. Query retention
  • U. Usability/UI at every level
  • I. Information defined organization
  • D. Drill down, vertical architecture

L – Levels: RDB, data(OO), info, ontologies – Applications should be composed in levels. Each level can interact with all of the levels below it. And each level has its own design principles, for example SOLID at the data/object oriented level. One level is the correct level for each need.
I – Indirection of fields – members of objects should be identified by information, not code, i.e. not within structs, or as class members.
Q – Query retention – The queries used to create the result sets need to be stored as part of the result sets.
U – Usability/UI at every level – each level should be usable by the one above and uses the ones below. Debugging/user_interface can start at any level. Levels identify themselves.
I – Information defined organization – Information should be used to distinguish ‘subclassing’, ‘graph relationships’, ‘hierarchies’, ‘DB relations’ etc. rather than inheritance. Each layer still has its ‘native’ organization, however.
D – Drill down – Vertical architecture lets the user drill down into program in order to interact with it, rather than having architecture composed of UI/midtier/Data layers through which the user generally can not drill. Expansion may consist of vertical integrated plug-ins.

The relationship of SOLID and LIQUID

SOLID is used as part of LIQUID wherever the needs of the software are best handled by object orientation. This results in hybrid object/info oriented systems. So LIQUID is not really an alternative to SOLID. LIQUID incorporates SOLID when needed. SOLID does not apply to information oriented work however, except at the data(OO) level:
S – single resp   NO , resp encapsulation MAYBE, the problem is that information gets used everywhere and if you change it you must change it everywhere
O – open/closed   NO , [INHERITANCE NOT USED, CLOSED FOR EXTENSION, SUBCLASSING SIMULATED BY INFO INTERPRETATION], open for modification, leads to perfection
L – substitution  N/A, [INHERITANCE NOT USED, CLOSED FOR EXTENSION, SUBCLASSING SIMULATED BY INFO INTERPRETATION]
I – interfaces    NO , prevents drill-down
D – dependency inversion MAYBE – normal dependency of high level upon lower-level components limits the reuse opportunities of the higher-level components

The second I in LIQUID refers to the following situations

– graph     – A -> B [is a]
– subclass  – A .. B [inherits from]  information ‘subclassing’ through information parameters and implementation
– hierarchy – A: -B  [has a]          information ‘grouping’ through some generic structure
– list      – A – B  [after a]        lists orderable by information
– RDB       – A -< B [relates to a]   table/column is addressable and characterizable, and business-rule-able

Data and Information

Object Orientation is the best when working with data. There is a huge difference however between data and information. When working with information,  it seems that the opposite of SOLID applies, escpecially the O)pen/closed L)iskov and and I)nheritance parts. Part of the reason for this is that with information oriented code, relationship is managed through data rather than through code organization. Members of objects/classes and sub-classes of classes are the organizational elements in OO that I am referring to. LIQUID stands for Levels by need, Indirection of fields, Query rentention, Usability/UI for each level, Information defined organization, and Drill down through vertical architecture.

Balancing the Abstract and Concrete

The Decision about when to go Abstract and when to go Concrete exists in programming. Here are some random notes on the subject.

Deciding on the Level of Abstraction:

Too much concrete gets in the way of the abstract – abstract allow you to build code at a high level. Think of concrete as sludge. If you have too much of it you can barely move. Too much abstract buries the concrete – concrete allows you to maintain code. Think of abstract as water, too deep and you drown in it. The analogy of an effective system is wading through waist high water. The water(abstract) is fluid and flexible, the bottom of the lake (concrete) is muddy but supportive and can be walked on as long as the water is deep enough so that it supports most of your weight. Too little abstraction and you get stuck int he mud. Too much abstraction and you are over your head.

A common example of too concrete is using low level string manipulation rather than using regular expressions. In this case one regular expression can often replace dozens of lines of code. A common example of too abstract is ASP.MVC. It is difficult to trace where things are happening and step through.

Complete Abstract Systems

Go abstract if you can build a ‘complete’ abstract system. The challenge of complete abstraction is building a place to start debugging and maintaining code. A complete abstraction system has:

  • storage (the magic strings appear nowhere except in the storage)
  • admin (the magic strings are managed using an adequate admin system)
  • magic strings do not appear in code, enumerations or case statements
  • usage & middle tier & display
  • benefit (a clear benefit is evident for going abstract)
  • testing is adequate so that you do not have to dig down and debug much
  • debugging display – bugs tell you how to fix them
  • another clear benefit
  • you can work with both data and information
  • you can mix in concrete as necessary – including debugging ‘hints’.
  • There is a place to start debugging, maintenance, and investigating the code in addition to the UI and database layers.

Not Going All The Way with Abstraction:

  • if you see mounds and mounds of fairly similar code, your abstracting system is not complete.
  • if programmers go crazy trying to figure out how you are doing things, then your admin/concrete hints system is not complete.
  • if there is not ‘level above’, your system is not complete

Frameworks are an Abstracting System:

  • The framework is good if it does not stop you from doing what you want.
  • Concrete programming (without framework) is a concrete system.
  • You can mix the two.

Abstraction can be used to:

  • abstraction to ‘clean up the code’. This is often a good idea.
  • abstraction to ‘genericize the forms’. This is sometimes a good idea.
  • piling everything into one method or stored procedure, handled by parameters/queries. This is often not a good idea.
  • table name stored in a variable. This is useful only in a ‘complete’ abstract system.
  • case statements to set the value of table name variables. This is a bad idea.
  • constructing dynamic queries in .NET. This is sometimes good, sometimes not.
  • Using case statements where you could have just written two methods. This is a bad idea, let .NET do the work.
  • using case statements to set a value where you could have just gotten a value out of a dictionary. This is a bad idea.

small abstractions are sometime quite useless, for example With/using.

Synching the Level of Abstraction:

Too many case statements may indicate over-abstraction combined with under-abstraction. The abstraction levels between things that need to work together have not been kept in synch. Things sometimes get out of synch when you are trying to do information things in code rather than data.

Debugging and Abstraction:

When you don’t abstract, you can go from the UI or the Database layer and figure things out. When you do abstract there is no obvious place to start.

The challenge of complete abstraction is building a place to start debugging and maintaining code.

The Stagnation of Technology

In The golden quarter Michael Hanlon wrote “Some of our greatest cultural and technological achievements took place between 1945 and 1971. Why has progress stalled?”
Here are a few clues:

  1. The missing zero – classical civilization failed to advance for a thousand years because of the inability of these societies to use the zero. The zero is the basis for place value. and place value is the basis for modern mathematics. The missing zero lead to an inability to efficiently represent mathematical thought.
  2. Evidence of a missing link – The software development industry shows that there is a missing link from business to artificial intelligence. If this missing link existed, a large percentage of business programming would include artificial intelligence of some sort rather than the significantly less than 1% we have today.
  3. The wrong turn for computer programming/science probably came about about sometime around 1971. Computer science is probably the bedrock foundation technology for the modern world. Just as with classical civilization the inability to materially represent thought could be the reason for our stagnation. My candidate for the wrong turning is hard coded context with the Algol 68 ‘struct’.

I believe that the creation of the ‘struct’ in Algol 68 was a major wrong turning in computer science and that it stifled innovation and progress in computer science and thus the rest of the technological field. One way of looking at the struct is – hard coded context for data. Hard coded means the data contexts (members) are part of a program rather than something that can be processed by a program.

The struct has become utterly predominant in computer science, with an almost complete inability to use other more advanced approaches in modern software development. Modern frameworks, OO and SOA have institutionalized the struct.

Without approaches other than the struct, we can not write information oriented software. Information can be defined as that which has meaning. One major form of meaning is context. We are unable to embed soft coded meaning/context in computer code. Since meaning/context is hard coded we are unable to write programs that process it. Without the ability to process embedded meaning we are only able to go so far.

A few more primitive approaches still hang on such as SQL and functional programming, but these are being progressively more and more sidelined. They are being sidelined because they have severe problems with
1) storing meaning and more importantly 2) being easy to develop with and 3) the ability to cover the needs of customers and applications.

The focus of my research right now is building a framework that will allow us to work with a soft-coded version of the ‘struct’.

Signs of problems with the object oriented paradigm

Signs of problems with the object oriented paradigm.

1. Code is too bloated. code is too repetitive, code is too low level.
2. Businesses to not reuse application code, applications get rewritten instead. Businesses have contempt for continuous improvement of the application because they know they will replace it in a few years.
3. The next level of development (information) is flat out missing on the internet, the fields/professions/job titles for people that do this work do not exist.

Not that the object oriented paradigm is wrong, it just gets used too universally for too many problems
it is like a golden hammer. It works fine for data. It suffers when used for information.