why is magic incompatible with civilization 4x type games?   Is it not just another resource or tool available to a player or AI?

A unit on the world map and a magic spell both have potential to impact the game.  They are both resources.  You have to adjust the game to take into account the potential for these variables.

Your X's must be different from mine if Endless Legend doesn't meet the definition.  I guess everyone brings their own definition to the table.

I ment Brads proposed game and not EL.

Ooh - sorry Ashbery!  Just ignore me. 

Brad's game doesn't sound as 4x, but sounds really cool.  I think I'll reinstall Elemental and play it from that perspective and see what happens. 

I completely agree... Magic or high-tech, no difference.

As for MP, it is a retreat. A lot of games of late are MP only, which is the result of AI coding being hard if you want to make it good. I never buy any game that does not have a single player gameplay. Sure, some games are far better in MP (Frozen Synapse), but SP is a must for me.

When I first read this, my initial reaction was to discount it.  There is nothing unfair about magic:  you only have to balance it, like everything else.  But to do so is to discount the years of Frogboy's proven track record.   So it's got me to thinking what he is really trying to say, or what perspective it is he is coming from.   

I remember an earlier analysis of magic way back even before computers really become common household items.   Wargame designers were discussing D&D-style wargames (i.e. board games, counters, dice, all that), and the biggest observation that stood out to me was, they found D&D-style warfare to be a lot more like modern warfare than medieval warfare.   If you've got a bunch of unarmored magic users who can throw balls of lightning from their hands at each other, that really is not much different from early modern warfare where everybody's ducking behind cover shooting at each other.    Now bear in mind:  this was before computers.  So there is no AI -- only two wargamers playing each other.  

So that leads to the next step in the thought process that is commonly missed:   ducking behind cover.  We keep thinking sword-wielding fighters will take the front, while magic-users stay behind and cast spells; all out in the open.   But guess what:  even our own military commanders missed this concept for real.   The entire Napoleonic era was spent using ritarded military tactics of amassing musket formations out in the open and two columns facing each other, 20 yards apart.  Granted, it wasn't THAT ritarded at first:  muskets were wildly inaccurate, and the only known way at the time to deal with it was to amass concentrated fire.   But gradually rifled bores came out, and by the late American Civil War the whole idea of shooting each other out in the open just became completely stupid--and that's BEFORE automatic weapons came out.   THAT was unfair--to the attacker.   The advantage to defense became a bit lopsided.   What are we to say if we write a Civil War game w/ an AI, then?   That your game is unbalanced??   No--because that was the reality.   

Therein lies the reality for you when writing a fantasy game:   you have to flesh out all the ramifications on the world you're making when you introduce fantastic elements into the equation.   If your battlefield is dominated by missiles of lightning flying all around, do you expect your troops to be wearing chainmail and plate?   If you have some powerful magic crystal sitting in a dungeon somewhere that could destroy the whole world if it gets in the wrong hands, will you send a party of 6 in?   Don't you think it more plausible that your sorcerer who knows this would alert your entire civilization, and the entire race would send in a whole army?    If orcs are constantly raiding the countryside, do you seriously expect your peasants to just sit around in shacks by themselves farming their old farmland?   That's why the Wild West had forts--except these aren't Indians, who were sometimes peaceful and in fact were sometimes in the right--these are ORCS.   

If you introduce some potentially unbalancing magic to the game, first:  that's okay.  It happens.  Automatic weapons are unbalancing, nuclear weapons are unbalancing--and those aren't even magic.   That really happened.  But you have to adjust to the new reality--and that means you have to adjust your play to the new reality, adjust the AI to the new reality, and evolve entire civilizations to adapt to the new reality.  I don't think it's an "unfair to the AI" problem:   I think it's that your new WORLD is not completely coherent yet.

Magic is as fair as you design it to be. Once you strip away the layers, it's really only dice rolls and numbers, just like anything else in a strategy game.

Agreed, but I guess I don't understand what exactly Frogboy is saying, and he has a successful track record creating AI, stories, and entire worlds.

I guess it's that the game community already has a software library of cookie-cutter 4X strategy games, with source code for cookie-cutter AI for it.  But once you introduce magic, how much can you reuse your cookie cutters?  You're not making cookies anymore.

you're making magical cookies.

Nicely put, especially this observation at the end of your post, tetleytea: 

 Consistency in spells, weapons, magical schools and effects would go a long way to making the game feel more like a world and make the mechanics more intuitive.   Creating balance and tension between might and magic, life vs death, quantity vs. quality in troops, spears vs. mounts etc.  are all of the things that make a good 4x fantasy game so exciting to play.  

i.e. Something as simple as terrain effects.  In MOM there were creatures who had mountaineering and flying that could reach a town behind a mountain range.  In FE:LH, they removed the ability of an earth mage to raise up the earth and shield his people behind a lofty bulwark of stone as the AI had pathfinding issues. If the AI had units it could create with the ability to climb mountains, or a spell it could cast to grant that ability, that would be a more interesting way to overcome the obstacle.  It becomes another choice to make in the game.  Do I spend mana to create an army with mountaineering, do I take up a unit perk slot and add construction costs?  Do I ignore that enemy until I research air elementals and attain summoning mastery and I spam a whole army of flying death? Do I cast Fly on a galleon and send a flying ship to carry troops?  Do I just hit his mountain stronghold with malicious spells, do i bribe a rival faction with mountaineering to take them out?  Move and counter-move...

As work begins/continues on E:2015 it would be great if some of these basic issues are revisited, thus capturing some more of that MOM feel of truly open sandbox, anything can be overcome if I find the right combination of spells, units and magic items.

Magic and technology can be completely overpowering. The problem is that creating an AI to utilize the magic nor that is it complete unfair. It is "unfun" unless you are the dominating party. Say you go nukes and the other team doesn't.... then you complain at the other team not doing a good job against your nukes. Make an AI to counter something that cannot be countered once obtained.

Try and defend you land with horse cavalry when the other side has tanks. The point is the AI should have gotten tanks... or your other argument would be the AI is cheating and has tanks and I don't unfair / unfun. People in essence want a balanced unbalancible game. I want the ability to tip the scales, but then it is a race to who gets it first.... not going to be fun.

The point is the type of AI to counter each of these tactics, albeit would be difficult to create, would not yield a fun game in the end. The asymmetric experience is the only way to accomplish the problem.

Let us add reasonable magic... well then what would be reasonable. Unfortunately it's magic and therefore anything should go and if not then it won't be fun. You want big effects like Curgen's Volcano to decimate the entire map easily... or I want the ability to change the entire map to ICE. These effects can be rather game-breaking and only fun if you are controlling the situation. That I think is more the point. The AI will not complain too much if it is losing, but a player would. The AI will handle being on the other end of the beating stick better than a player would.

The other issue that magic brings to the table is that it is not well defined what should be possible and should not be possible. Fun doesn't always equate to realism.

so many people are just dismissing AI as some impossibly hard problem

the AI in 4X games is terrible because developers are a combination of lazy, poor, and incompetent

if you put skill levels on a scale between 0 and 3000 (eg. some type of elo rating), then the AI you see in 4X games today is way less than 1000

getting it to 3000 might be hard when you want a game to run on a laptop, but that doesn't mean it can't play at 2000 or 2500 which might be good enough to solve any design problems... and for those top 1% of players the AI can cheat with minor bonuses instead of +300% faster/cheaper everything

We'll have to agree to disagree there, Biz.  AI is a very hard problem.   I would in fact argue that the conspicuous lack of good AI in complicated games is due to the fact that AI doesn't get enough respect.   AI is what powers things like facial recognition on Facebook or musical recognition on Shazam (...which, incidentally, runs on smartphones).   Look at all the manpower thrown at a chess AI, and chess is a far simpler game than pretty much any strategy game out there today.   It is the subject of PhD research--and those guys get hired out of college.   The gaming companies usually don't pay enough to recruit all the best AI coders. 

There are lot of good points here.  In most fantasy settings, magic takes the place of technology.  When games are set in a more modern or futuristic setting, the AI typically doesn't need to understand how to use the technology, as it's more or less baked into the core mechanics of the game; how you attack/defend/produce, etc..  It's all utilized in the background as you just play.  In a fantasy setting, you could make mana the most important resource and have it affect more stats passively but with magical flavor text for the player.  But, if it was just that, magic doesn't feel as mysterious because it's just like technology we already have (oh look, a mage staff with an ACOG scope and an extended power crystal).  

Of course, I may be rambling a bit.  I think what Brad meant is you can't necessarily just slap some magic on Civilization and call it a day.  You have to bake magic into the core mechanics and make it special (not just stats but spell-casting also) so the AI can deal with it and so the player intuitively understands how and why the AI is doing what it does (and so the player can do it too).  

That's a ridiculous statement, and rather insulting to developers who might read it.