Back to Squawk list
  • 57

Boeing outsourcing of software development

Submitted
Much has been written about the 737Max's problems, but this topic has hardly been mentioned. The author writes for Bloomberg, but this was published in the Sydney Morning Herold. (www.smh.com.au) More...

Sort type: [Top] [Newest]


Aspectcarl
Carl Richards 17
We outsource sw development to India and I have experience in Telecoms and Automotive, these are the problems I have generally observed, some of which are not necessarily Indian engineer related:

Indian engineers tend to be well educated and intelligent, but the vast majority are fresh from university and have little grounding in the final use.

There is often a reliance upon machine written software.

Communication between teams is poor and assumptions are continually made.

Well written specifications can be difficult to find.

Reviews at all stages happen infrequently.

Contracts exist where suppliers can provide untested software.

Management reaction is to have a part of the team write a software quality guide for the supplier :(

Timescales are generally unrealistic

When something in the build script doesn't work, someone fudges it with best intentions to get an overnight build out, and then the headaches start.

My employer has embarked upon another project under sub contract to another country that is well developed and same issues exist with software quality.

NASA/MIT documented many of the pitfalls of software development many years ago but it appears many organisations prefer to start from the ground up and make proprietary mistakes.
w2bsa
w2bsa 8
Being a retired IT professional some of these problems are not new, the big one being well written specifications. Sometimes they are almost nonexistent. All of these exist in ALL software development at some point. The thing that prevents that to some extent is to always do software development on site where the work can be supervised by the client using IT professionals that are hired and paid by the company the software is being written for. It seems to me that Boeing didn’t do this or they were cutting costs by not using redundancies for the MCAS. There is also the problem of too much dependence on automation.
patpylot
patrick baker 23
how many more nails in the coffin of boeing management is required before complete housecleaning , including walking them out the door without benefits and pensions? This is leadership, or is it shopping for the cheapest way to do thing, regardless of meeting safe standards?
Pachypodium
Including walking them out the door and into prison. Their management led to the deaths of 346 people. They should stand trial in Ethiopia and Indonesia.
DaveRK
DaveRK 15
From 2010 to 2017, I was working on the hardware of a emerging developmental software-based electronics system.
Anytime the software dev. folks made a change, they spent days "regression" testing the software.

I bought a new motorcycle in 2014, the 1st year for an "in-dash" infotainment system.
There were many silly little problems and every time a SW update was done, something else would go awry. After being assured by the SW Dev. folks the problems were all in the SW coding and lack of proper re-testing, I eventually called the manufacturer of the MC and then the radio manufacturer.
I assume I became such a pest, eventually the radio manufacturer told me "The software is being developed by a third party company in India, we have no control over that" WTH?
I thought about finding that company and contacting them then realized "Hell I can't get a simple problem solved when I contact an off-shore call center about anything".

IMO, In The Long Run, all this "Outsourcing" regardless of the product is costing these companies money not saving it.
Quirkyfrog
Robert Cowling 24
My sister worked for a company that depended on controct programmers from India.

She said it was a 'shit show'. The 'screw ups' were incremental. She was the lead for their QC team in New Hampshire, and she was always stunned at the errors that the Indian programmers introduced into their software.

Once, someone apparently just grabbed a 6 month old code base, did their change, and submitted it for review. She called, practically in tears. 'How in the hell!'

So management got a 'Great Idea(r)', we'll send our QC team to India to meet with the programmers assigned to the project, and maybe their dedication will rub off on the constant cycle of programmers she had to deal with. I mean, she was in THEIR time zone. She got the the office while it was still dark, and babysat the programmers in India, in real time.

So, she and her team, several quit at being 'asked' to go to India, and she sat with their management, and met with the 'lead programmers', and experienced the smells, and dysentery of India.

Half the 'team' in New Hampshire quit, the other half said, in no uncertain terms, they would NOT GO TO INDIA, and the problems continued.

So, handling programmers almost half a world away, with a definite language barrier is going to be MASSIVE, and even more so in an industry where if they don't look at every character, and test that 'software' every way that it could fail, they are stupid, and the lives that were lost in the two crashes were negligent HOMICIDE, and the management of Boeing needs to be frogmarched out of their office, and turn in their golden parachutes at the door.

Hired India programmers is a crappy way to insure your product doesn't kill people. My sister said that they spent likely more testing their product than they would have if their IN HOUSE programming team has done the damn job in the first place.

Likely, Everett thought that they could trust the programmers, the hired guns, they forgot that the minimum wage India programmers aren't going to die when that software fails, and the plane succumbs to gravity. People die management goes 'Oh, crap', and they keep testing fate...
djames225
djames225 6
Sounds so much like the shit shows of off shore call centers. It's fine if they want to get cheaper help, but for crying out loud, get the help that knows WTH is going on, how it's going on and why.

Something in that report has me puzzled...Boeing said the "cockpit warning light" was an "option"...so how is it an issue if it was working according to their "plan"? "The Chicago-based planemaker also said it didn't rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn't working for most buyers." Won't work if it was an "option"
Irishmh
Here’s the opinion of one long experienced Sw Dev - albeit with experience in slightly different industries.

The entire situation is indicative of the risk acceptability vs bottom line costs that prevails these days. Overwhelming importantance is placed on costs & timelines vs quality. It’s basically become acceptable if not expected to have flaws, recalls & the like - no matter how critical the nature of the deliverable is. It’s now common place to not take the proper time, pay low wages, & not support the concerns & undesired analysis results found & communicated by reputable employees that will not accept anything less than what is needed to ensure software & products are designed, developed & function properly Before they are delivered/implemented.

Many of the latest “fads” of software development demand these kinds of issues/concerns/incorrect results be voiced. Yet when those techniques are implemented & issues are questioned/identified in some instances management doesn’t want to hear anything negative & occasionally even crucifies that “whistle/blower”. Thus software ican be installed & products delivered that may well have flaws. And those flaws can inconvenience or even hurt/maime/kill people as well as cause untold property destruction - plus take considerably more work & time to resolve.

The legal results may be deemed “better/justified” from a financial aspect when weighed against missing delivery dates or the bottom line costs.

This is a worldwide issue. At some point we as a country & even race need to demand & be able to expect better. Quality MUST prevail - at least when lives are at stake - if not always...
z1hou
G R 3
What makes people think that not having Indian programmers will not result in something similar. The article is not clear whether they developed software for MCAS.
And if you pay $9/hour, you cannot expect anything stellar. It is called "work to rule.".
Heck, from what I understand, most software even at the kernel level is developed by India/Indians. Just take a look at all the Silicon Valley companies. All of them.
I think the US has steadily lost its lead in technology implementation. China, Japan, North Korea, Germany, Norway and Sweden are implementing technologies innovated here. And the innovation part is also moving away. I'll just leave it a that.
Cansojr
Cansojr 1
This is a comment I would make siriusloon to read in light of his self righteous ability to slander third world nations. Good Comment.
jpcooper
Peter Cooper 12
I'm only glad that Preacher 1 is no longer alive to see what Boeing management have done to the company he so valued and whose aircraft he delighted in flying for so many years.
Greg77FA
Greg77FA 16
Get what you pay for. Have to ask Board of Boeing, is it worth betting the company on cheap, non-US Labor? Sad that Indian programmers are working on this advanced plane - anyone in the software world knows that outsourced Indian labor is mediocre at best. Yes, you will have cheaper labor, and more people on your project, but it takes 2-3 solid people to keep a very close eye on them, and there will be multiple iterations to deliver anything of quality.
mbrews
mbrews 9
the MAX program managers may now realize - " You get what you underpay for "
JOHNMONDAK
JOHN MONDAK 10
I have seen this happen in the telecom world. All I can say is: You get what you pay for"
ToddBaldwin3
ToddBaldwin3 7
Part of the problem, in my humble opinion, is summed up in this line "The Chicago-based planemaker...."
kalirick
Rick Hunt 5
Everyone is dumping on Boeing and running to Airbus. Has everyone forgotten how a known pitot tube issues and associated flight control software brought down a huge plane and killed bunch of people? How soon we forget.....
n9341c
n9341c 4
To all of you guys complaining that it was cheap offshore contractors that caused the 737 MAX to crash, did you actually read the article or was it just too convenient to overlook the actual content of the article?

"Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March.

The Chicago-based planemaker also said it didn't rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn't working for most buyers."
djames225
djames225 5
I'm sorry, but when you read this line "The Chicago-based planemaker also said it didn't rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn't working for most buyers." and this line Boeing also has disclosed that it learned soon after MAX deliveries began in 2017 that a warning light that might have alerted crews to the issue with the sensor wasn't installed correctly in the flight-display software.", it makes you wonder how much truth Boeing is saying or just trying to cover their pervberial butts. Boeing earlier said the "warning light" was an option to airlines, now it's a software issue?
And then there is this "Rockwell Collins, now a unit of United Technologies, won the Max contract for cockpit displays, and it has relied in part on HCL engineers in India, Iowa and the Seattle area. A United Technologies spokeswoman didn't respond to a request for comment." The report from the Ethiopian crash also stated there were anomalies happening with the cockpit displays. From what I have seen and done, if I program a part or 2, that programming must be able to communicate with all other "links" in the system, and if it's messed up, it messes with everything else.

So to answer your question, yes I actually read the article.
michel09
Michel Lahaie 7
The MCAS was not ‘linked’ to the crashes. Enough of the PC verbiage. It was the cause of those deadly accidents. The program was dependent on only one Angle of Attack indicator.

As a pilot and owner of several aircrafts, including turbine driven ones, I would never buy nor fly anything without redundancy.

Who wrote the program is hardly relevant. Who approved the certification is the guilty party.
JimHeslop
Jim Heslop 3
Nicely put. But Boeing kept that info (MCAS) and its related lack of redundancy from the pilots in the first place. I just can’t fathom how management is still there and not bending over in the prison showers.
ghstark
Greg S 2
Correct, nor is their any evidence that MCAS software was buggy anyway. It just appears to be a correct implementation of a terrible design.
djames225
djames225 4
It actually was "buggy"...please read my response to allan leedy below.
Funny thing...I get down voted for a correct comment and he upvoted...so much for an equal system I guess.
milehighou
milehighou 3
Outsourcing and paying the software guys 9 bucks an hour, LOL. I was at Target the other day, and saw a help wanted sign...starting pay was $13/hr. People at Target are being paid more than the ones Boeing trusted to code flight control software. Unbelievable.
Jeraboam
Jeraboam 2
Read the preceding comment by Allen Sjolie: $9 an hour is a good wage in parts of India just as $40 an hour is a poor wage in parts of the USA. It is a separate issue from the quality of the workforce. India has developed an outstanding university system to create top of the line engineers in every field, as good as and even better than many comparable universities in the USA.
mjsraman
Shankar Raman 3
All the intellectuals sitting at desk and blaming Indian engineers have not mentioned a single word on why FAA regulations wes bypassed in some cases by Boeing. I have worked with Bad Engineers in the US too. Getting paid fat salaries for owning boats and aircrafts but carrying no logic in head while coding. So stop blaming and look into the root cause of bypassing regulations.
SpaceRanger
I have a new mantra: "If it's a Boeing, then I ain't going!"
agsjolie
Allen Sjolie 2
Folks - I don’t disagree with your comments as long as their based on verified factual information. I think we can all agree that much of what we read is based on sloppy reporting. My comment was merely based on my hesitancy believe a news story.. How well verified is the story? I don’t think the contractor being based in India suggests inferior intelligence. A $9/hr contract employee in India is similar to $60/hr employee in USA. I do agree that it is ultimately Boeing’s responsibility to validate the contractor work. Every company has people of varying capability and intelligence. Because none of us no the individuals that actually worked on the project , we can only comment on our own assumptions based on a reporters possible research. I’m not ready to throw all contractors under the bus because they don’t live in the USA possess a Boeing badge.
ianmcdonell
ian mcdonell 1
A very detailed article thanks
pthompso99
Kind of reminds me of current state/federal "efforts" to allow importation of drugs from overseas. We only take a total of 5 drugs in our household, all generics. We only bought at local pharmacies. However, when I did a little sleuthing, I found out 4 of the 5 were made overseas!! So much for being local.
paultrubits
paul trubits 1
A pharmacist here. Almost all generics are now made in India. We have FDA inspectors there. Last I heard they only had 4 inspectors. When you are overworked, you will probably cut corners. Sounds like they let SWA slide a little.
pthompso99
Yes. Hmmm. Four inspectors. ll of India. Cutting corners. And ultimately, that's the US' fault.
raleedy
ALLAN LEEDY 1
Is this even relevant? I thought we were dealing with a flawed concept, not a mistaken software implementation.
kd7eir
Jim Myers 0
It's the SOFTWARE that is taking over and crashing the planes. Have you not comprehended anything you've read about this?
raleedy
ALLAN LEEDY 2
Yes, and by design, not because of some coding error.
djames225
djames225 1
It was a coding error and should have been caught..what is suppose to be drilled into our heads with aircraft, and many other items? Redundancy, redundancy, redundancy. That redundancy does not include coding a computer/computers to only trust 1 sensor and nothing else.
raleedy
ALLAN LEEDY 3
That was the design.
djames225
djames225 1
No that was not the original design. The original design was taking both sensor readings and only applying a minimal, I believe 2.8 degree augmentation of the stabilizers. HOWEVER due to coding error(s) the system kept getting more and more "bogged" to the point only 1 sensor would tell the computer and it would trim, wait a bit, trim, wait a bit, trim, wait a bit...THAT was never caught and it was allowed to go to production. They should have wiped the computers and re-did the whole code!
tremy0207
Terry Remy 1
Why is Dennis Muilenburg still CEO? Who is head of QC? who Is head of IT? You get what you pay for. Lives have been lost to save dollars. If the Benjamins guides your decision, then make sure it is not at the cost of lives.
FLYINGSC0T
FLYINGSC0T 3
I'd be happy to see all the Max's make their way to the boneyard in Arizona - there might also be room there for the CEO, head of QC and the head of IT!! Sorry, but I feel so strongly against the "homicide" that has been committed.
conx
charlie lange 1
Boeing needs to clean house at the top. Then it needs to move headquarters back to Seattle. Then it needs to bring everything back to northwest Washington. Everything. Boeing used to be the best until they started farming everything out for political gain and cost cutting. Glad they made their bigly bucks. Too bad it was on the backs of so many people...some living... and too many dead.
FLYINGSC0T
FLYINGSC0T 1
Since the two aircraft disasters I was of the opinion that those who programmed the MCAS software had no real knowledge of what they were programming and the implications if it went wrong. With Boeing outsourcing the software development and effectively "deskilling" could only ask for trouble. Now Boeing has got it and at great expense for the lives lost. If I was the CEO, I would resign !!
rgraham11
Robert Graham 1
Well, it seems Boeing leadership was in the midst of saving lots of money when everything 737 Max went south. There are 346 people who were sacrificed to those savings and the decentralization of control. As stated by other commenters, Boeing's upper management needs to be swept away and common sense and safety restored.
agsjolie
Allen Sjolie 1
Why do so many make the assumption that a software development contractor equals low quality engineering work?
wlhamaty
Luke Hamaty 4
Because many have experienced that this is often true. It isn't completely the contractor's fault: successful use of contractors requires considerable oversight.
Quality work might cost more up front, but costs less in the long run. Management, however, may not care about the long run, even when they clearly should. They may not understand the difference between quality and shoddy work. There is also a tendency to view software development as a mechanical task, which it is not.
If you ask a contractor to do something stupid, they will often just do it rather than tell you it is stupid. It is hard to find people who will take ownership, understand the design and specification, and stick their neck out if they see a flaw. It is much easier to just do as you are told. Many contracting firms encourage that sort of thinking. Management that rewards compliance over diligence will not achieve quality, outsourced or not.
In the case of MCAS, there are indications that those who DID raise their hand were ignored, certainly not rewarded.
Aspectcarl
Carl Richards 4
You need to work with them, the bell curve applies as much to software engineers as any other facet of engineering.

You can subcontract a task, but you cannot subcontract responsibility, accountability and any form of ownership.

I've met and worked alongside some cracking Indian engineers and I count them as good personal friends, but I've a lot of mediocre engineers who need hand holding and constant follow up, and whilst they hold a degree they have no concept whatsoever of the final product. The good ones busy themselves asking questions and finding out.
bigkahuna400
bigkahuna400 1
BOEING CEO has to go...it is time....Maybe shareholders need to hold a " No Confidence" vote and boot him and others...
Quirkyfrog
But as long as those shareholders think they will get big money, they will allow him and his tear to remain.

They pay so little on taxes on their profits, they can afford to lose some value. The idea is 'maximizing profit', and as long as the current CEO is doing that (and screwing unions) they will be happier than pigs in...
flyingphil42
Phil Ford 1
I totally agree!
akebonolove
akebonolove 0
Nobody cares, just look at the share price...
paultrubits
paul trubits 2
The stock is down almost $100. The $$$ guys and gals care.
djjamar
Jamar Jackson -6
That plane is a piece of 💩
shenghaohan
Shenghao Han 0
Don’t judge the book by the bad Amazon reviews. There are is a reason why there are more 737 than A320s exists.

That said, not matter no good the plane is, it will still become the victim of the shortest board in the barrel... the faulty software.
djames225
djames225 2
I believe he was referring to the MAX edition when he said "That plane is a piece of 💩 (a bunch of symbols)"
btweston
btweston 1
This is not just a 737, pal.
ghstark
Greg S 0
Outsourced software is very hard to integrate into existing software, and very hard to enforce quality standards on. Furthermore, the software industry understands the quality implications of outsourcing to Indian companies, you get terrible quality every time without fail. It is actually hard to believe Boeing would do such a thing. Boeing's management team is a disaster, and the Board of Directors that picked them is also a disaster.
sonolisto
Why not just change the name of the 737 MAX as Donald Trump said?
angelruki
a p 2
Sure, they should name it Boeing 737-DISGRACE.

Once the name, and only the name, is changed you and the Donald can fly it together...Good luck, honestly.
btweston
btweston 1
I’m honestly not sure if you’re serious.
shenghaohan
Shenghao Han -2
Who else think even their ECO stepped down won’t do much for current affair?

Login

Don't have an account? Register now (free) for customized features, flight alerts, and more!
Did you know that FlightAware flight tracking is supported by advertising?
You can help us keep FlightAware free by allowing ads from FlightAware.com. We work hard to keep our advertising relevant and unobtrusive to create a great experience. It's quick and easy to whitelist ads on FlightAware or please consider our premium accounts.
Dismiss