Decision Making and Computational Creativity

When I think about where automated/computer decision making exists in this stack, it is in level 6 – computational creativity. You see, level 5 – artificial intelligence analyzes the world around it, and level 6 synthesizes things from its understanding. One of the things that can be synthesized is decisions. Level1&2:data. Level3&4:information. Level5&6:understanding. Level7&8:reproduction.

When building endemes by hand you are making decisions, so this form of building is a creative action. Endemes systemetize the construction of information, so endeme building can systemetize creativity.

Each Level Has its Own Native Structure

I will start with an example.

Here are examples at each level to handle a user-customer relationship:

  1. Level 1 – RDB: A join table:
    user customers
  2. Level 2 – OO: A few classes depending on use and need:
    a user table with a customer objects list or a customer id’s list depending
    a customer class
    a user class
    a ‘client’ data structure that contains both objects (references to them)
    a user-customer relationship class
  3. Level 3 – IOP: Three endeme sets defining user-customer fields:
    a UserIdentity set
    a CustomerIdentity set
    A relationship set
  4. Level 4 – KR: Two nodes and a relationship edge:
    customer node
    user node
    relationship edge

Native Organization Schemes

  • Level 1 – RDB – relations
  • Level 2 – OO  – hierarchies
  • Level 3 – IOP – matching/(or memberships?)
  • Level 4 – KR  – graphs

For matching, each piece of data is part of whatever it matches well. For membership, each piece of data is part of multiple things.

The First Three Organization Schemes Are Implemented Through Four Types of Tables:

  • data tables   – usually have a large number of rows focused on data
  • lookup tables – usually have a smaller amounts of rows focused on data [OO]
  • endeme tables – usually have a small number of rows focused on ‘type’ [IOP]
  • join tables   – plumbing used to connect data tables to tables in a n to n relationship [RDB]

Given the needs of various part of a project, work with one or more of these for different purposes.

Resulting in These Structures:

  • hierarchy – oo is especially good with these [like lookup tables]
  • join      – uniquely relational              [join tables]
  • search    – generally information oriented   [endeme tables and endeme structures
  • data      – everyone works with data to some extent
  • graphs    – above the level of database tables?

Then adding two more types of tables:

  • Endeme implementation tables – endemes
  • Graph implementation tables – knowledge representation


These two new groups of tables implement endeme and knowledge representation structures (information structures) in a database. They generally set somewhat apart within the database in their own little engines.

Wisdom is a Computational Creativity Function

Wisdom is a Computational Creativity Function. The computer has to create what it is going to do by applying its understanding gained through artificial intelligence.

An Example:

  • Data: (Raw) Red,, v2.0
  • Information: (Meaning) South facing traffic light on corner of Pitt and George Streets has turned red.
  • Knowledge: (Context) The traffic light I am driving towards has turned red.
  • Wisdom: (Applied) I better stop the car!

The Example Converted to Levels:

0.    Raw      Data
1.    Stored   Data
2. -> Meaning  Data types and objects and hard defined structural interpretation
3. -> Meaning  Information with structural (and conceptual?) context as meaning
4. -> Meaning  knowledge representation with relational context to user
5. -> Context  knowledge and what various outcomes and actions mean
6. -> Applied  wisdom – stop the car!
Level 6 is computational creativity. The computer has to create an action.

Context and meaning are both built up through multiple levels rather than just being at one level each.

An Inner Platform is a Good Thing?

In figuring out how to work with information inside a computer program, I am working toward building an inner platform. The thing I have to watch out for is an anti-pattern called the “Inner Platform Effect”. The inner platform effect often starts out with a generic database that gets added to and enhanced until it becomes a replica (usually a poor and incomplete one) of one of the systems that it is built on top of. Most often the kinds of inner platforms built are replications of relational databases and operating systems.

It is possible to build an inner platform that replicates the capabilities and organization of a relational database. It is possible to build an inner platform that replicates the capabilities and organization of an operating system. It is possible to build an inner platform that replicates the capabilities  and organization of an object oriented software development system.

In data oriented software development the inner platform effect is an anti-pattern. The Inner Platform Effect – “The Inner-Platform Effect is a result of designing a system to be so customizable that it ends becoming a poor replica of the platform it was designed with.” This “customization” of this dynamic inner-platform becomes so complicated that only a programmer (and not the end user) is able to modify it.

Three things tend to lead to the creation of an inner platform

  • Users want something that they can modify themselves without having to have a programmer involved.
  • Programmers want to build something very generic because they do not know what precise needs their user have.
  • Programmers may realize deep down that the way forward in software development is to build a level above the level they are currently working. This may be the real reason programmers keep building them and suffering the consequences. Without information structures and theory, you simply can not build an inner platform that has added more capability than a database or object oriented approach. Any or all of these may be in play.
  • Programmers look at repetitive field handling code and think to themselves that there must be a way of abstracting it. You see (to oversimplify the situation) each column in a table ends up being a member in a class. A whole bunch of members tend to look the same, get set, changed and erased and displayed the same way, and the thinking is that there really should be a generic abstracted way to handle them.

Avoid the Inner Platform Effect

When you are building an inner platform there are some things you can do to avoid the inner platform effect:

  1. Do something the underlying platform cannot do or can do very poorly. This is my goal – to build an inner platform that works with information as well as data.
    Working with information is something that a database or an object oriented programming can barely do if at all. Usually when someone attempts to use one of these technologies for information, they end up with a snarl of excessively complex, nearly unmaintainable code.
    Endemes provide work-with-able information about the items in the ‘inner platform’. Information about the items is what inner platforms are normally missing.
  2. For simple information: avoid objects and attributes in the endeme tables. have only endeme strings, labels and ids
    • have the endeme id’s in the tables (objects) directly
    • the endeme is the value
  3. for endemized attributes/data:
    avoid objects in the endeme tables, each endeme for the data is its attribute
    – the endeme is the attribute, the endeme’s value is the data, the label is the name of the attribute
  4. for endemeized objects: This would be a generic database and full inner platform.
    I don’t know how to avoid building an inner platform here. I need much more experience working with 1,2,ad 3. above before I can hope to work with endemized
    objects without suffering the inner platform effect. The trick is, that in order to get this experience, I need to build a bunch of inner platforms, have them die their expected gory death due to the inner platform effect, and eventually figure out how to build a true information programming platform that does not suffer this.
    need to use level 4, can not do this properly until level 3 is well understood
    this would build an actual inner platform.
    – endeme lists would avoid the problem of not using the database for some of its excellent features.
    this is because the database would need to support custom indexes, (since information has no order) and functions and triggers to power those indexes, (since the database does these very well)
  5. Level 4 programming will be based on an inner platform. We need to build an inner platform for level 4 information programming. This will allow us to use knowledge representation in programming. Note to self: look to see if there are any successful knowledge representation inner platforms out there. OWL comes to mind.
  6. AI programming will be based on the knowledge representation programming and is possible once an inner platform is built for level 4 programming. The inner model has a column for id, key/field, and value.

More thoughts on avoiding the inner platform effect

  • how do I avoid the inner platform effect?
    • keep the inner platform really sparse in features
    • do a hybrid system
    • have columns for id(must have), field(in only its information format), and value(must have)
      • access by information, not ‘table’ and ‘key’
      • but an endeme list has endemes with labels, access by information, the endeme makes it different
      • have no search attributes, put these in regular table columns
    • don’t let a user modify it, only for programmers
    • build a solid infrastructure to hide the implementation details
    • implement in really slow steps, taking years and learning how to do each step well before moving on to the next one
  • how do I use the RDB for what it’s good for?
  • the question is what can I do with information?

Building a Successful Inner Platform

The inner platform effect is an intuition by programmers that we need to build a level above the data levels. However without the information oriented programming theory to base it on, inner platforms just generally only implement things that already exist. Thus they usually fail to build a level above data levels.

I want to take the inner platform above both the database, the OS and the object oriented program level. The challenge is to have the inner platform reach out to the rel platform it replaces to use the performance facilities of the real platform below. The inability of an inner platform to do this is the most common downfall for the inner platform.

An inner platform for a database:

  • table
  • column
  • row
  • relation
  • pk
  • value

An inner platform for object orientation/xml/json:

  • class
  • member
  • object
  • type
  • collection

An inner platform for an operating system:

  • path
  • folder
  • filename
  • parent
  • child

Generic Data Models

– object     attribute     value

From C2: Definitely not an anti-pattern, and it can often greatly simplify table design for some corner scenarios. Apart from giving rapid prototyping benefits it also helps with handling ‘jagged’ data. Instead of introducing null columns using GenericDataModel enables us to save only relevant data thereby actually decreasing complexity. Reporting and manual labor against the table can be handled via views as mentioned above. not

Entity–attribute–value model

  •     The entity: the item being described.
  •     The attribute or parameter: typically implemented as a foreign key into a table of attribute definitions. The attribute definitions table might contain the following columns: an attribute ID, attribute name, description, data type, and columns assisting input validation, e.g., maximum string length and regular expression, set of permissible values, etc.
  •     The value of the attribute.

From Wikipedia: As noted above, EAV modeling makes sense for categories of data, such as
 – clinical findings, where attributes are numerous and sparse.
Where these conditions do not hold, standard relational modeling (i.e., one column per attribute) is preferable; using EAV does not mean abandoning common sense or principles of good relational design. In clinical record systems, the subschemas dealing with patient demographics and billing are typically modeled conventionally. (While most vendor database schemas are proprietary, VistA, the system used throughout the United States Department of Veterans Affairs (VA) medical system, known as the Veterans Health Administration (VHA),[1] is open-source and its schema is readily inspectable, though it uses a MUMPS database engine rather than a relational database.)

We Need an Inner Platform for Information

We need an inner platform because the db and the language do not provide what we need and we need a platform that will.
the problem is based on members

The DRY Principle and Information


In software development, DRY stands for Don’t Repeat Yourself. Conventionally this principle gets implemented by frameworks like ASP.NET MVC and Ruby on Rails by having a single source for column/field names, either the database or a class.

Framework Challenges

The challenge with these systems is to manage the sources. This requires high integration, and makes it very difficult for these frameworks to play with others. They are also very solidly based in a data oriented approach, meaning that they institutionalize hard coded context. With hard coded fields even when there is only one source for each, framework APIs become inflexible.

An Information Based DRY Framework

An information approach would treat each field as something to be processed and characterized rather than just as a field name and a type and other hard coded attributes. Information could provide flexible relationships between fields, flexibly created new fields, and control for the source of the field names, labels, types, and parsers.

Interacting with Another Framework

An endemized English vocabulary could be used to come up with an automatic signature for each of the columns in a database, each of the fields in entity framework, and each of the members of various data classes. These could be glommed. I have an endemized signature vocabulary for about 5000 words already available. Adding synonyms and two word definitions, and including another 1000 very common database column words could push this to 10000. Relationships could be identified from class memberships, operations and stored procedures. In this way an endemized framework could interact with MVC and Ruby on Rails without a lot of code written (once this framework was written).

An API to Anything

A further DRYing of code could happen by making the endemized framework a source. Once the framework above works, then this step becomes simple. And the endemized information framework can be a conduit to the other sources. We can build an API for interacting with the endemized framework giving other systems access to fields and rich metadata about fields in apps and databases. Endemizing provides flexible metadata, so there could also be an API to adapt the generalized framework to specific sources.

The Dreams of Computer Programming

The dreams of computer programming:

  • The dream of writing a true AI program.
  • The dream of writing programs to meet our needs and desires and to provide benefits to us, for example including increasing productivity.
  • The dream of extracting information from data by writing a program.
  • The dream of a general purpose problem solver that for example can fly your car, cook your eggs, play chess with you and write a symphony etc. the key idea being that it is able to tackle whatever problem you might set it.

We have not met these dreams yet. Instead, true AI is still very far away. Only a fraction of our needs and desires get met. Instead of writing programs we use big honkin’ data mining and big data tools. We still write specific programs to do specific things.

I think the problem is that software development has reached a glass ceiling because we have not developed level 3 programming yet.

Level 3 programming requires three things

We need general programming, domain programming, and level 3 programming frameworks:

  1. General programming is what we do right now with object oriented coding and object oriented code reuse.
  2. Domain programming has to do with mixing our programming with concrete exececutable models of combinable, permutable concept lists focusing on the concepts and concept groups of the domains. i.e. endemes. This is just another definition for endeme sets.
  3. A level 3 programming framework allows us to execute the executable models of combinable, permutable concepts. Rather than just complaining, I think it is important to also provide solutions to resolve what I complain about. Therefore, I am working on a level 3 framework. My latest framework is posted here.

What is Programming?

Programming is all about defining the meanings of words and using them to do things.

Our next step is to write programs that handle the meaning of words or something analagous to that. I could call this ‘Managing data using word meanings’. ‘Finishing the processing of the meaning of words’. or ‘Finishing the processing of words’.

I am thinking about the need to make a case for the existence of level 3. Very little work gets done on this level as far as I know. So maybe level 3 does not exist.

My Case for Level 3

My case for its existence is – there are bits, numbers, words, and sentences

Level 4 is all about relationships
level 3 is all about meaning
level 2 is all about data
level 1 is all about storage

level1 1 2 and 3 all seem to have something to do with the meanings of words:
level 1 provides the storage
level 2 provides the processing of words using hard coded context, for its precision.
level 3 completes the meaning of words
level 4 puts the words in context

Maybe Typing is the Problem

maybe compilers are not the problem. maybe typing is the problem. Not just strong typing, any typing. We need something more flexible, and yet still processable. Xml is a form of context, however it has the hard coded typing problem. Why is hard coded typing a problem? Because hard coded context itself can not be processed. Is that the real problem?

The idea of meaning for data and context to manage it

Context is critical. relationship is critical but it is not primary. The meaning of a word by itself is critical. We need to learn how to handle words. We have made words hard processable. Now we need a layer on to op the hard processable words that finishes
the job of lower level programming.

What will it take to finish the job of meaning aside from relationship?

1. context processing – contexts are a sort of relationship but what I mean by context is context by class and table.
– the ability to process that context is important – is this level 3 or level 4?
– the part that is level 3 is the part that manages the contexts of levels 1 and 2
! we need a context manager.
2. endematic meaning – the meaning of the word (mostly) by itself – in other words the definition of a node in a graph.
3. the meainig of words
the endeme set to be applied – is this level 4?
4. a system to store a words meanings and endeme sets

endeme – enumerated meme – this is the definition of an endeme in A.I. terms.