Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support network emulation options #1098

Open
markjmeier opened this issue Aug 1, 2019 · 11 comments
Open

Support network emulation options #1098

markjmeier opened this issue Aug 1, 2019 · 11 comments
Labels
evaluation needed proposal needs to be validated or tested before fully implementing it in k6 feature new-http issues that would require (or benefit from) a new HTTP API

Comments

@markjmeier
Copy link

It would be useful if different network conditions could be emulated during a test by introducing some latency and limiting bandwidth available to the virtual users. This helps to cover more real world cases for testing. Predefined options are nice but will need maintenance over time, probably easiest to just allow direct setting of latency and bandwidth in a test.

@na--
Copy link
Member

na-- commented Aug 1, 2019

This shouldn't be too hard to do, though I'd prefer if we do it after we have custom user-definable transports. There isn't a separate issue describing that feature, I'll create one in the next few days and link to this as well as other things that depend on it, but the gist of it is described in #1045 (comment).

Regarding the implementation details, I remember that I've seen a library for this in Go, but I couldn't find it. I only found these seemingly unmaintained ones:

Worst case, we'd have to roll our own, but even that shouldn't be too difficult.

@na-- na-- added evaluation needed proposal needs to be validated or tested before fully implementing it in k6 feature labels Aug 1, 2019
@dipeshlshah
Copy link

My company is building app which will receive only 3g / 4g user traffic.
I do see some article from 2014 stating that Loadimpact was supporting this feature https://k6.io/blog/mobile-network-emulation-the-key-to-realistic-mobile-performance-testing-feature-release

We need this feature in order to simulate mobile user network traffic. We use Annual Pro k6 subscription to load testing and planning to upgrade in future once we grow. Can you prioritise the fix/feature ?

@imiric
Copy link
Contributor

imiric commented Jul 23, 2020

Wouldn't users be better off by using an external proxy to introduce latency and network issues, instead of it being part of k6 itself? This could be done by something like mitmproxy or muxy.

Sure, we could custom fit the solution to work for k6, and the UX would be better eventually, but just like for the case of CPU/RAM monitoring inside k6, I feel this would be better left out. Existing tools cover these and any other complex scenarios, and users would be better off learning these tools instead of using a k6-only solution, even if it ends up being easier to use.

@na--
Copy link
Member

na-- commented Jul 23, 2020

@imiric, while there are certainly workarounds and other tools, I'm pretty sure we'll have to build this functionality in k6, for a few reasons:

  • as you said, it will be simpler (though, for that to be true, I'm 99% certain it will have to be done in the new HTTP API)
  • it will allow users to have exactly the same functionality in both local k6 run load tests, as well as in k6 cloud ones, which a proxy won't do
  • given the new k6 v0.27.0 scenarios, it will allow users to simulate a mix of traffic speeds - with a proxy, it will be difficult to emulate, say, 80% of VUs having broadband connections, 10% having a good mobile connection and 10% having a spotty 2G connection or something like that

So, yes, @dipeshlshah, we plan to add this feature in k6, and in our cloud service, but I can't give any promises on when it will be available yet. In order to implement it in a way that allows our users to make the most of it, we need to implement some prerequisites first.

@na-- na-- added the new-http issues that would require (or benefit from) a new HTTP API label Oct 18, 2021
@amandahla
Copy link

Any news regarding this issue?

@na--
Copy link
Member

na-- commented Oct 31, 2022

No, sorry. We will share updates here whenever there is progress.

In general, k6 currently lacks the concept of a "network transport" and/or "dialer" objects that you can tweak (e.g. by artificially throttling its speeds or doing other sorts of network emulation) and that you can then pass to a "HTTP Client" object to use. We also lack this "HTTP client" concept, which is the reason this issue is kind of connected to #2461, though it's probably a good idea to keep them separate... At the moment, every VU has an implicit global network transport and an HTTP client that are not very configurable 😞

k6/js/runner.go

Lines 155 to 200 in 2fe2dd3

dialer := &netext.Dialer{
Dialer: r.BaseDialer,
Resolver: r.Resolver,
Blacklist: r.Bundle.Options.BlacklistIPs,
BlockedHostnames: r.Bundle.Options.BlockedHostnames.Trie,
Hosts: r.Bundle.Options.Hosts,
}
if r.Bundle.Options.LocalIPs.Valid {
var ipIndex uint64
if idLocal > 0 {
ipIndex = idLocal - 1
}
dialer.Dialer.LocalAddr = &net.TCPAddr{IP: r.Bundle.Options.LocalIPs.Pool.GetIP(ipIndex)}
}
tlsConfig := &tls.Config{
InsecureSkipVerify: r.Bundle.Options.InsecureSkipTLSVerify.Bool, //nolint:gosec
CipherSuites: cipherSuites,
MinVersion: uint16(tlsVersions.Min),
MaxVersion: uint16(tlsVersions.Max),
Certificates: certs,
Renegotiation: tls.RenegotiateFreelyAsClient,
KeyLogWriter: r.preInitState.KeyLogger,
}
// Follow NameToCertificate in https://pkg.go.dev/crypto/tls@go1.17.6#Config, leave this field nil
// when it is empty
if len(nameToCert) > 0 {
nameToCertWarning.Do(func() {
r.preInitState.Logger.Warn(
"tlsAuth.domains option could be removed in the next releases, it's recommended to leave it empty " +
"and let k6 automatically detect from the provided certificate. It follows the Go's NameToCertificate " +
"deprecation - https://pkg.go.dev/crypto/tls@go1.17#Config.",
)
})
//nolint:staticcheck // ignore SA1019 we can deprecate it but we have to continue to support the previous code.
tlsConfig.NameToCertificate = nameToCert
}
transport := &http.Transport{
Proxy: http.ProxyFromEnvironment,
TLSClientConfig: tlsConfig,
DialContext: dialer.DialContext,
DisableCompression: true,
DisableKeepAlives: r.Bundle.Options.NoConnectionReuse.Bool,
MaxIdleConns: int(r.Bundle.Options.Batch.Int64),
MaxIdleConnsPerHost: int(r.Bundle.Options.BatchPerHost.Int64),
}

@kedare
Copy link

kedare commented Nov 15, 2022

I'm also interested by this :)

I developed a custom backend for file upload and I wante to test how the system would react:

  • Under high latency
  • Under high packer loss
  • With very slow uploads
  • With upload stopped in the middle (but not interrupted)
    All this with a lot of concurrent uploads at the same time.

So far I think I will try to combine k6 and https://github.com/tylertreat/comcast

@pablochacin
Copy link

The xk6-disruptor extension allows the simulation of different conditions such as latency or even random errors, in the responses from HTTP and gRPC services.

One interesting attribute is that it allows injecting these conditions not only in the target of the test (for example, a front-end) but also in any other service this system under test depends on.

Presently it supports only protocol (L7) fault injection for Kubernetes services, but in the roadmap we are planning also network level disruptions.

@imiric
Copy link
Contributor

imiric commented Jun 8, 2023

Thanks for the heads-up @pablochacin! Great stuff! 🎉

Correct me if I'm wrong, but xk6-disruptor injects these network conditions on the SUT itself, right? According to the discussion above, the consensus is that this is something that k6 should simulate internally, so that a) it would be agnostic to whatever backend it's testing, and b) would work for any supported protocols now and in the future.

I tend to agree with the approach that this logic should live external to k6, but I also see the benefits of having it internally. I'm wondering what your thoughts are on this.

@pablochacin
Copy link

pablochacin commented Jun 8, 2023

Hi @imiric

In most tests, k6 is interacting with an endpoint which I would call the SUT. Simulating network conditions between the SUT and k6 is in general less interesting. If somehow you add more latency, k6 will see more latency. If you inject an error, k6 will see the error. The only thing you are doing is measuring the impact of these conditions on the user experience (for instance, the time to load one page, or the time to upload a file).

As someone also mentioned, you can test some error conditions such as what happens if there is a network error while unloading a file to a backend.

All of the above is valuable but I think it covers a very narrow set of use cases.

With xk6-disruptor you can inject these disruptions between any two components of the application and see how the application reacts to those conditions. Then you can evaluate how latency between say the frontend and some backed service impacts user experience.

You can actually also inject these disruptions on SUT, so you can simulate the scenario described above.

In summary, both make sense, but in my opinion, doing the fault injection in k6 is more limited in terms of what can be tested. That said, it is true that it is less intrusive and can be applied to any SUT regardless of how it is deployed.

EDIT: I made the comment above ☝️ thinking mostly about API tests. But now that we have k6-browser, the use cases for k6 fault injection are more interesting because we are testing the application running in the browser, not only the requests to the backend. However, I'm not sure how we could inject those faults in the browser given that it runs as an independent process.

@hash004
Copy link

hash004 commented Jun 22, 2023

Related feature request: grafana/xk6-browser#887

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
evaluation needed proposal needs to be validated or tested before fully implementing it in k6 feature new-http issues that would require (or benefit from) a new HTTP API
Projects
None yet
Development

No branches or pull requests

8 participants